Subscripts/indexing in general

Phuket2

Look this is not a bug report. But I am having some minor problems with the lack of retrieving objects via index.

For example I am using a framework for creating lists in UI. (Pythonista iOS App). It has like a middleware class to retrieve/delete/re-order data etc. The list supports sections and records.

So I get a callback that generally includes a section and a row. My idea is to use tables as sections and those records in the table as rows generically. It works ok, but I have to end up doing things like -

def num_records(self, section):
        with TinyDB(self.db_fn) as db:
            tbl = db.table(list(db.tables())[section])
            return len(tbl)

Firstly because I cant get a table by its index, so then because Tables() returns a set, i have to convert it to a list so i can get its value by an index. Section is just an int(index).

Again, to get the tables name by index i have to do -

def get_section_name(self, idx):
        with TinyDB(self.db_fn) as db:
            # db.tables() returns a set
            return list(db.tables())[idx]

I had a strategy to get/remove records by index before I started using sections (single table). I am not sure what I will do now, but i will be working against the nature of TinyDB.

Anyway, i will get by. but it just seems to me that some indexing capability on some internal objects could be very useful. Even if they are not super fast. You normally need to take a hit on things that are generic.

Also starts to make more sense if you have a DataProvider class that is talking to a TinyDB built class, meaning you could swap the backend quite easily given you provide the same api.
Well at least in my mind.

Phuket2

Phuket2
Just to give a better idea, the below is where I am at so far. I just show the TinyDB class and the datasource class that I am mapping to the TinyDB class. It's not complete, but its getting there.

The Move callback maybe quite tricky to implement. I am sure its possible, maybe just slow. But on small datasets will not see it I think.
Also, the cell creation in the callback 'tableview_cell_for_row', is not correct yet. I will pull that out and make cell externally. It can be generic for simple things, but for what I have in mind it will be handled separately.

Just posted, to give a bit more context to what I am trying to achieve.

class TinyDBBackEnd(object):
    '''
    For whatever reason, I decided not to leave the database open.  
    '''
    def __init__(self, db_fn, *args, **kwargs):
        self.db_fn = db_fn
        
    def add_record(self, rec_dict, key, section_str='_default'):
        with TinyDB(self.db_fn) as db:
            tbl = db.table(section_str)
            tbl.upsert(rec_dict, where(key) == rec_dict[key])
              
    def get_section_tbl(self, db, section):
        return db.table(list(db.tables())[section])
            
    def num_records(self, section):
        with TinyDB(self.db_fn) as db:
            return len(self.get_section_tbl(db, section))
            
    def get_indexed_rec(self, section, idx):
        with TinyDB(self.db_fn) as db:
            tbl = self.get_section_tbl(db, section)
            doc_ids = list(tbl._read().keys())
            return db.get(doc_id=doc_ids[idx])
            
    def del_rec(self, section, idx):
        with TinyDB(self.db_fn) as db:
            tbl = self.get_section_tbl(db, section)
            doc_ids = list(tbl._read().keys())
            tbl.remove(doc_ids=[doc_ids[idx]])
            
    def num_sections(self):
        with TinyDB(self.db_fn) as db:
            return len(db.tables())
            
    def get_section_name(self, section):
        with TinyDB(self.db_fn) as db:
            return list(db.tables())[section]
            
            
class MyDataSource(object):
    def __init__(self, db_fn, *args, **kwargs):
        self.db = TinyDBBackEnd(db_fn)
        
    def tableview_number_of_sections(self, tableview):
        # Return the number of sections (defaults to 1)
        return self.db.num_sections()

    def tableview_number_of_rows(self, tableview, section):
        # Return the number of rows in the section
        return self.db.num_records(section)

    def tableview_cell_for_row(self, tableview, section, row):
        # Create and return a cell for the given section/row
        rec = self.db.get_indexed_rec(section, row)
        cell = ui.TableViewCell()
        cell.text_label.text = rec['timer_name']
        return cell

    def tableview_title_for_header(self, tableview, section):
        # Return a title for the given section.
        # If this is not implemented, no section headers will be shown.
        return self.db.get_section_name(section)

    def tableview_can_delete(self, tableview, section, row):
        # Return True if the user should be able to delete the given row.
        return True

    def tableview_can_move(self, tableview, section, row):
        # Return True if a reordering control should be shown for the given row (in editing mode).
        return True

    def tableview_delete(self, tableview, section, row):
        # Called when the user confirms deletion of the given row.
        self.db.del_rec(section, row)
        tableview.reload_data()

    def tableview_move_row(self, tableview, from_section, from_row, to_section, to_row):
        # Called when the user moves a row with the reordering control (in editing mode).
        pass

msiemens

Hey Phuket2 , just for my understanding, are sections numbers? Cheers!

Phuket2

msiemens , yes they are numbers. However they are just indices. I realise there are some faults in my logic trying to suppport the DataSource middleware as I have. So maybe my post is invalid.

But I still posted it, as I still feel being able to access elements by an index could provide some flexibility in making generic solutions for some use cases. I am sure there would be a performance hit in some cases for using methods that used indexing. But if we are talking about small datasets, could prove to be insignificant compared to the ease of implemention in your app.

I have mentioned before, I am not very experienced. So I am just posting what makes sense to me. I accept I could be totally wrong.

msiemens

I see two options here to make it work better:

Use numbers as table names. As far as I can tell, this should work just fine. You then can use db.table(number) to access your section.
Use a more specialized tool than TinyDB. You could create your own fork that fits your use case better than TinyDB. No one will get mad at you ?

In general I think this is a really rare use case so optimizing TinyDB for it would in some way affect everyone else who doesn't need indexing (and if it's "only" code size, which matters too). On the other hand, with your own fork or implementation you're free to change all the details as much as you want ? And of course you can copy parts of TinyDB that are still useful to you like the Query language.

I hope, this helps a little.
Cheers!

Phuket2

msiemens , thanks for your reply. Using the table numbers would not work as in this case the table name is also used as the section name. But regardless, I understand your answer. It makes sense to keep all optimise for the many use cases, not the edge use cases. I am not sure how your user base is using TinyDB, thats why I put this forward.

You're right, I could switch to some other lib, but I wont. I am just a hobbiest programmer. So what's important for me is to get to know one lib well and be comfortable with it and use it for most if not all my modest persistent data needs. If I come across a problem that is not 100% match for TinyDB, for me its better to put a little more effort to get it to work rather than switching technologies. I am sure a fulltime/professional developer would take a different approach.

But again thanks for your reply. Sorry for my delayed response.

Phuket2

I know this matter is basically closed. I just re-visited the indexing idea today. I have done very little with "get_item" & "set_item" dunder methods. When I have tried it, its been difficult to understand all the side effects.
Anyway, today I just gave it a go, using a custom Table to see if I could get slicing to work on a TinyDB database. Only getting, not setting. In this case it seems fairly trivial, because I am using a list of doc_ids, the list is handling all the slicing. Sorry if this seems obvious. To me it was not. I am not even sure if the below is problematic (regards breaking on slicing). I realise the speed is debatable. On small data sets, speed does not seem to be an issue. I would still be interested in any thoughts.

Edit: I should say the below is for python 3.6 , should be easy to convert to py 2.7

from tinydb import TinyDB
from tinydb.database import Table

from faker import Faker
fake = Faker()

def make_dummy_data(num_items=100):
    '''
    Create some dummy data to use
    returns a list of dicts built up using Faker
    '''
    return [
            {'idx': i,
             'first': fake.first_name(),
             'last': fake.last_name(),
             'age': fake.random_int(min=18, max=106)
            }
            for i in range(0, num_items)
           ]


class TableIndexAccess(Table):
    '''
    Crude attempt to add accessing of docs by index by overriding
    __get_item__ in Custom table.
    '''
    
    # don't need the __init__ just put it there to demonstrate. Not using it!
    def __init__(self, storage, name, cache_size=10, *args, **kwargs):
        super().__init__(storage, name, cache_size, *args, **kwargs)

    def __getitem__(self, idx):
        ids = list(self._read().keys())[idx]
        ids = ids if isinstance(ids, list) else [ids]
        return [self.get(doc_id=id) for id in ids]


if __name__ == '__main__':
    db_fn = 'table_test.json'            # database filename
    dummy_data = make_dummy_data(500)

    db = TinyDB(db_fn)                   # create TDB instance w/filename
    db.purge_tables()                    # clear all the tables, just for testing

    db.table_class = TableIndexAccess    # Tables created will use TableIndexAccess
    tbl = db.table('data')               # create a new table
    tbl.insert_multiple(dummy_data)      # add the dummy data to the new table

    # try some slicing
    print(tbl[0:5], '\n')                # first 5 docs
    print(tbl[::len(tbl)-1], '\n')       # first and last docs
    print(tbl[0:50:3], '\n')             # try with step param in slicing
    
    db.close()                           # close database

Phuket2

After this post, I was thinking rather than subclass the Table, just write the get_item func above in a func called say my_get_item and then with a valid table reference then do something like tbl.get_item = my_get_item. I could not get it to work, I also tried doing it with the db object also. I am not skilled enough to know why this does not work. But it would be a nice shortcut vrs having to subclass the table.

Phuket2

Another thought, is that table.get() could also support a param of list of doc_ids as some other get methods support. This would negate a list comprehension using get. Even if this use case is not smart, it seems like that the get method could always work with doc_ids (list of doc_id) rather than a mix support in the api. To me , this would be more consistent