reads the whole data file twice

mghasemi

I am writing a custom storage for tinydb and I noticed that upon initialization, the TinyDB calls the read method of the Storage twice. Is there any way that initialization can be done in the first read and avoid reading the whole data for the second time?

msiemens

Hey mghasemi ,

thanks for the hint! I've changed TinyDB to only call Storage.read once during initialization instead of twice, see commit b100617. I've added a reminder for me to release this in a new version of TinyDB in two weeks, if nothing else comes up in the meantime. Until them you can use the latest development version from GitHub if this change is critical for you.

Cheers!

mghasemi

Awesome! Thanks.

mghasemi

Seems that every time I want to access the data file through a custom storage, it still calls the read method twice, either at initiation or at any time I want to retrieve something out of db.
I will look into this later, as it is not an urgent issue which causes weird behavior, but just a performance issue.

msiemens

That's weird. Did you install the latest version from the master branch?

mghasemi

Yes I did.
Just put a simple print at the top of the read method of the custom storage and initiate a TinyDB object, the print statement appears twice.

msiemens

Funny, I even added a test that confirms that read is only called once during initialization – see test_storages.py#L138-L167.

(Also, by the way, I made TinyDB only do one read when writing data (insert, update, …) instead of two like it used to).

mghasemi

Hah!
You know, I am reading the list of tables right after initiation, and that's why it calls the read method twice now.
I wonder, if it is necessary though.
Thanks a lot.

mghasemi

So, to avoid multiple reads and writes, I use a buffer variable as an attribute of the storage, and update the buffer upon each call to the write method. Then, to commit changes, I have to call the close method that actually flushes all modifications to the data file. This resolves performance issue.

By the way, since I started using tinydb, and realized how customizable it is, I use it in various cases like handling csv, excel, json files as well as a substitute for sqlite that stores data in encrypted format ... .
I haven't noticed any performance issue, until once I throw a rather big excel file at it. But tinydb already has a workaround for that ?

msiemens

mghasemi So, to avoid multiple reads and writes, I use a buffer variable as an attribute of the storage, and update the buffer upon each call to the write method. Then, to commit changes, I have to call the close method that actually flushes all modifications to the data file.

This reminds me of TinyDB's CachingMiddleware which does basically the same ?