msiemens Would it be possible for you to convert all numbers during import of your CSV?
Hi, just updating that I'm taking your advised route now, by using Pandas read_csv instead of csv.Dictreader. Here's the relevant code snippet, there were a few things to take care of, so sharing in case its useful to someone:
dtype_dic= { 'service_id':str, 'end_date':str, 'start_date':str, ... }
feedArray = pd.read_csv(feedfile , na_filter=False, dtype = dtype_dic).to_dict('records')
db.insert_multiple(feedArray)
Explanation:
na_filter=False: tells pandas to treat blank cells as empty strings instead of the default NaN.
dtype = dtype_dic: tells pandas explicitly to cast the specified columns as string. This is for cases where the data is in numberals like '20180405' but you want it stored as string only. The good part: it won't error out if that column isn't present in the csv being read. So that lets us define one big mapping json to cover all our csv's.
.to_dict('records') : converts the pandas dataframe to a flat list of dicts. 'records' is the parameter telling it not to do indexing etc, just keep it flat.