memory issue with tinydb

jwchi

hi, I wonder if I am using tinydb the correct way. what I have is a bunch of series/array objects that I want to store in tinydb (about 1000 series/arrays with each of them containing about 3000 elements, total size on harddrive about 40Mb). basically what my program does is to get the array, turn it into a Python dict (serialization), and insert it into tinydb. I observe that the memory of the process keeps growing and growing until it crashes with a system error ('cannot reserve memory' error), which looks a lot like a memory leak.

Is there a known issue with how tinydb handles memories/serialization? This has been rather frustrating as I like tinydb and wish to make it work, yet it doesn't work with a larger dataset.

I'd appreciate any comment/trouble shooting advice.

jwchi

forgot to mention - i was using tinymongo as the accessor to tinydb..

it turns out that the memory leak seems to be happening with tinymongo (not tinydb itself to be precise). maybe i shall create a script to recreate the issue.

msiemens

Hey jwchi ,

were you able to find the memory leak in TinyMongo or report it to the project? I just tried a scenario similar to yours and it completed without issues. That being said, if you work with this amount of data, you'd probably be better of using sqlite or something similar as TinyDB's performance is definitely not tuned to cases like this.

Cheers!