Karanveer Singh

Oh. I hadn't considered that.

I've now updated to

records = {r["timestamp"]: r for r in df.to_dict("records")} # filter the records by timestamps not already present in db data_dict = [ rec for rec in records.values if rec["timestamp"] not in set_of_timestamps ]

I do have a unique timestamp index, but before I insert, I have the following python logic to filter only those timestamps which are not present in the collection.

set_of_timestamps = {entry["timestamp"] for entry in collection.find()} # filter the records by timestamps not already present in db data_dict = [    rec    for rec in df.to_dict("records")    if rec["timestamp"] not in set_of_timestamps ] if data_dict:    logger.info(f"Inserting to db...")    collection.insert_many(data_dict)    logger.info(f"Completed inserting {len(data_dict)} record(s) to db.")

This is a sample document.

{     "_id": {    "$oid": "61f6ba8d23135306891e8240"  },     "timestamp": 1640908800,     "open": 47110.3,     "high": 47112.2,     "low": 47097.03,     "close": 47097.03,     "volume": 0.15180822,     "time": { "$date": "2021-12-31T00:00:00Z"  },     "symbol": "BTCUSD" }

Is there some other way for me to filter timestamps to those not present in the db?

Hi,
This has been bothering me for weeks now.
I have a python script that periodically queries timeseries data from a crypto api, and inserts it into my mongodb collection.

Every now and then, seemingly at random, the collection.insert_many function will fail with the error message shown below. I formatted the json so it's easier to read.
The id field is not generated by me and is supposed to be created and managed by Mongo. So how can I prevent the collision on a field I don't provide?

pymongo.errors.BulkWriteError: batch op errors occurred, full error: {  "writeErrors":[    {      "index":0,      "code":11000,      "keyPattern":{        "_id":1      },      "keyValue":{        "_id":"ObjectId(""6255cce8bdfc280a8c8d6ee8"")"      },      "errmsg":"E11000 duplicate key error collection: crypto.btc_ohlc_minute index: _id_ dup key: { _id: ObjectId('6255cce8bdfc280a8c8d6ee8') }",      "op":{        "timestamp":1649786640,        "time":"Timestamp(""2022-04-12 18:04:00+0000",        "tz=""UTC"")",        "symbol":"BTCUSD",        "open":39993.64,        "high":40008.58,        "low":39979.97,        "close":40007.3,        "volume":0.0382197,        "_id":"ObjectId(""6255cce8bdfc280a8c8d6ee8"")"      }    }  ],  "writeConcernErrors":[],  "nInserted":0,  "nUpserted":0,  "nMatched":0,  "nModified":0,  "nRemoved":0,  "upserted":[] }

That's really helpful. I'm excited to dive into these topics and make it come together. Thank you.

Hi Alexander,

In my work I've encountered scripts and automations which are a collection of imperative functions which are called exclusively for their side effects.
They are composed entirely of I/O steps (fetching data from database, posting to another service, writing logs to file) which are also sequential in terms of time.

What strategies could be applied to begin to make these kinds of imperative programs more functional?

Congratulations and thanks to Vitaly for promoting Haskell and the book!

Hi Vitaly,

I'm a CS student in my senior year.
Like so many others, I've been drawn to Haskell for the beauty of FP and declarative programming, and am interested in blogging about it for my own growth and to help others.
It's a language you just can't get tired of coming back to because you learn more every time you return to it, and find a new stream of ideas waiting for you.

There are great resources to learn Haskell, and they're constantly growing but I wanted to discuss specifically which industry or types of projects you feel can increase adoption and interest in the language.

What aspects of the language would you like the community to highlight or blog about, that would soften the learning curve for newcomers from Python or Java?

Welcome Vitaly!

Haskell is a very unique language.
I've always enjoyed venturing into it, like a climber. One has to put some effort to gain some footholds, but the view it offers over the programming landscape is worth it.

I have the OCA Java 8 cert. I remember studying for it and really getting familiar with evaluating Java code in my head. About a week later I had a Java interview and got the job.

At that job there was a java enterprise app written by consultants with PhDs in math and CS who were not around to enhance the application. There was also no JavaDoc comments to speak of - no public classes or methods had documentation.

I was the only one on the team who had a Java cert so was given the task to enhance it with the new requirements. I did what I could and started reading. It was really challenging at times but after about 2 months, I was able to create enough of a mental model of the codebase, and how to use the classes and interfaces, to make the enhancement.

The OCA certainly made me familiar enough with Java such that syntax, and following along with code blocks was not a hurdle for me anymore. The main challenge was the semantics, the domain modeling, and design patterns which I had to really struggle to understand the how and why of it all. Those things the cert will not teach.

Does Go have its own version of proposed features that way Python has PEP and Java has JCP?

Yes. I think both books tackle all topics in a self contained manner.

You'd have an easier time with the second book if you're familiar with the content of the first book. Like understanding generics, classes, collections, interfaces etc.

I would classify the first book as the title suggests, it's core Java. The second book I'd classify as being modern java since it goes into the details of features of Java 8, 9 and 11.

Also, the first book has more imperative programming examples since streams and lambdas are not too prominent in the first book. But the second book introduces those concepts and adds some functional programming to the code examples.

At my new job, the backend is Java EE and my coworkers are all using JDeveloper. I'm not familiar with it and my experience with Java has been limited to Java SE and Spring development with Netbeans and Intellij. I can work in JDeveloper if I have to but I would prefer to use Netbeans or intellij. Is there a way to import a java ee project from JDeveloper.

There's basically two modules - an EJB project and a Web project which contains an Angular front-end build. Should I recreate each project separately in the other IDE? Any suggestions?

I've been learning JavaScript the past month and it's only re-affirmed my love for Java. I can't wait to learn about the new features.

This thread is great. I just discovered Java Streams last month and my brain was baffled and amazed with the ease and flexibility of functional programming. It set me off on an appetite for functional programming knowledge that has completely changed the way I write java programs.