Error while inserting string with Emojis

lbeato

Hi,

I'm trying to insert (using TinyDB method Insert) a string encoded in utf-8 that contains an emoji, but I'm getting this error:

UnicodeDecodeError: 'utf8' codec can't decode byte 0xd9 in position 47: invalid continuation byte

I have try every workaround found on the Web but can't seem to solve it. Can somebody help me?

msiemens

Hey lbeato ,

which version of Python are you using? Python 2? Python 3?

Cheers!

lbeato

msiemens Python 2

The tweet:

"RT @KydSplash: If it's ment to be, it will be ?"

lbeato

is there any solution for my problem?

msiemens

lbeato I'll try to look into this issue this weekend, but I can't promise it unfortunately

Cheers!

msiemens

lbeato Can you post the exact code you used to create the TinyDB instancde and to insert the string? I tried to reproduce this with Python 2.7 and TinyDB 3.13 and so far it works without issues…

The code I used:

# coding=utf-8

from tinydb import TinyDB

TinyDB('test.json').insert({'tweet': u"RT @KydSplash: If it's ment to be, it will be 💯"})

Cheers!

lbeato

msiemens This is my code. I read from a cvs and write to the Json:

dbTweets = TinyDB('/Users/lbeato/Documents/Doctorado/Codigo/Tweets.json', ensure_ascii=False)
with open('/Users/lbeato/Documents/Doctorado/Codigo/tweetslimpios.csv', 'rb') as csvfile:
   rows = csv.reader(csvfile, delimiter=',', quotechar='"')
   #This skips the first row of the CSV file.
   next(rows)
   for row in rows:
       dbTweets.insert({'id': row[0], 'tweet': row[4],'emotion1': row[5]})

By the way...your code works fine.

msiemens

lbeato Sorry for not replying yet. I'm a little short on time as I'm about to hand in my master's thesis in less than two weeks, but I'll try to look into this more closely this week!

msiemens

I found the issue ? With ensure_ascii=False the JSONStorage creates an ASCII only file. So you can either remove ensure_ascii=False to store the emoji as \ud83d\udcaf or use io.open to read the CSV file as an UTF-8 file and set the encoding for the JSONStorage:

import io
from tinydb import TinyDB

db = TinyDB('test.json', ensure_ascii=False, encoding='utf-8')
s = io.open('test.txt', 'r', encoding='utf-8').readline()
db.insert({'tweet': s})

(where test.txt contains the tweet and is stored as UTF-8)

Cheers!