Synfig/Krita/Kdenlive Animation Notes

From 2020-06-03

So you want to do hand-drawn animation on Linux and/or entirely with Free Open Source Software, and that animation involves lip syncing? Well, here's some notes as I work on my first decent sized animation in Synfig/Krita/Kdenlive and I get my pipeline ready to go for smooth production work on future animation.

• Get the latest development version if you're on Ubuntu, the version (as of 2020-06-03) in the Ubuntu repos has a bug where you can't enter in 0 for the angle of a rotation, and you also can't use Ctrl-A to select the contents of a field in the Properties of a group, and I'll risk any other weirdness in the app to be able to enter in 0 for the angle of a rotation and to use Ctrl-A.
  • If you want better (imho) Papagayo import that doesn't forcibly put a rest at the end of every word, grab this branch and build it. The build process for Synfig is very very nice, by the way! I wish all large apps were this smooth to build and rebuild.
• You should try out this version of Papagayo, cobbled together from various forks over the years, that is fast and supports single-frame words and has other nice quality of life improvements. Please help me get it building on other platforms!
• If you're running a modern Ubuntu and using the Synfig AppImage, there's probably gonna be a ton of fontconfig errors and the UI will look weird. Errors mentioning unknown element. Lots of them. This happens to me in Krita as well on another machine when run via AppImage, though this did not fix Krita on that machine. The solution is to run the AppImage and provide the system fontconfig libraries via LD_PRELOAD. The exact line I had to use is:

  bash LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libfontconfig.so:/usr/lib/x86_64-linux-gnu/libfreetype.so" \ Downloads/SynfigStudio-1.3.14-testing-2020.04.30-linux64-b2662.appimage

  Hunt down those two libraries in /usr/lib and replace the paths with what you have and the fonts will look all pretty and stuff. Or build Synfig from source and you won't have this issue.

• Inkscape .sif export is not well supported, even for things like rectangles. Synfig also doesn't seem to like Inkscape SVG files. This is fine, as I'm doing titling in Kdenlive and only importing hand-drawn bitmaps from Krita into Synfig.
• I was going to do the whole thing in Synfig, but I found a combination of Krita, Synfig, and Kdenlive is what you'll probably want. I envision a much larger, more involved blog post/video series once my Papagayo fix is in for Synfig. This also means separate Synfig files for, say, camera distances from things, so you don't need to worry as much about RAM usage in Synfig as you're making a lot of smaller scenes (exported at HuffYUV-compressed MP$ files) and assembline them in Kdenlive.
• By default, importing a bitmap is…wrong. The dimensions are all messed up. Edit > Preferences > Editing > Imported Image > Scale to fit canvas is what you want. This means you're importing in everything at full size, so export everything at full size from Krita. The exports are compressed PNG files, and disk space is cheap nowadays.
• It looks like you can easily build at a lower resolution, say 640×360, and then export to a larger resolution, say 2560×1440, and everything scales as it should. As long as the assets are high enough quality, of course.
• This pipeline works as expected because Synfig is storing references to imported images and not the whole image data:
  • Export rough animation frames in Krita
  • Import frames as image sequence into Synfig
  • Export cleaned up animation frames with same names in Krita
  • Reopen the Synfig file
  • The new, cleaned up frames will appear in Synfig
• Only put one voice into a Papagayo file, otherwise none of the voices will appear when imported into Synfig.
  • It seems like it's best to split the audio files into one clip per Papagayo file per character pose. In one scene, Bamboo will have two body/face positions, so I'll need two clips and two Papagayo files.
• The parts you need for your lipsync from Papagayo are listed on this lip sync chart with a few changes:
  • L is L, Th
  • etc is C, D, …

    Which means you'll, at most, need art for: AI, O, E, U, L, WQ, MBP, FV, etc, rest

• The best way I found to make the mouth parts and export them in a way that makes Synfig's handling of them less of a pain is:
  • Create a new group layer above the layer where the character's head is located.
  • Draw each mouth, in a new layer, with the layer named after the part.
  • Use Krita Batch Exporter to export the frames, one image per mouth layer.
    • The exporter can also export group layers, with visible layers merged down, so my standard inks → shading → colors layer setup will work just fine.
  • Import the Papagayo file into Synfig.
    • This is actually a link! The frames are reloaded if the Papagayo file is updated. (or if you modify Synfig's source code to process a Papagyo file differently!)
  • Import each mouth image individually, and drag the imported image into the appropriate folder in the Papagayo group object.
    • If the Papagayo file isn't openable in the Layer docker in Synfig, put the file in a Group, then remove it from the Group. I tend to put everything imported in a Group now.
  • BOOM, mouth animation.
  • Since Synfig supports Python plugins that work by manipulting a temporary Synfig (XML) document during an import, I'll figure out a way to import all of the mouth patterns for a Papagayo file somehow. Or, I'll just make a Python or Ruby script to modify the SIF file and put in the links via the command line. Yay XML! :(
• Below is a Python script to run in Scripter in Krita to create layers for each mouth pattern used in a Papagayo file. You only get the layers for the mouth patters you're actually using, which will save on drawing. Select a Group layer where the new layers should be deposited and run it. This sets up the layers I will typically use for an art piece: Pencils, Colors, Shading (set up Multiply blending mode), and Inks. This works fine in Krita 4.3.0:

snippet.python

  from PyQt5.QtWidgets import QFileDialog
  from krita import Krita
 
  KID = Krita.instance().activeDocument()
 
  active = KID.activeNode()
 
  file, _ = QFileDialog.getOpenFileName(None, "Create phoneme layers for Papagayo file", "", "Papagayo File (*.pgo)")
 
  # rest is the default when a frame doesn't have a specified phoneme
  phonemes = ["rest"]
 
  with open(file) as f:
      for line in f:
          line = line.rstrip()
          if line.startswith("\t\t\t\t"):
              _, phoneme = line.split(" ")
              if not(phoneme in phonemes):
                  phonemes.append(phoneme)
 
  for name in phonemes:
      layer = KID.createGroupLayer(name)
      active.addChildNode(layer, None)
 
      for artName in ["Pencils", "Colors", "Shading", "Inks"]:
          artLayer = KID.createNode(artName, "paintlayer")
 
          if artName == "Shading":
              artLayer.setBlendingMode("multiply")
 
          layer.addChildNode(artLayer, None)

All in all, the process was pretty decent. It took some time to figure out the Synfig way of doing things, but once I got that, the actual pipeline of Krita → Synfig → Kdenlive was very smooth. Expect something way more substantial describing the process in the coming months.