From bea7925208b5bcd87365c29a9a3e333cb41650f2 Mon Sep 17 00:00:00 2001 From: rtibblesbot Date: Sat, 4 Jul 2026 19:27:53 -0700 Subject: [PATCH 1/5] Lead onboarding docs with ContentNode(uri=) over legacy File/Node classes Rewrites the getting-started/tutorial examples and reference docs (nodes.md, files.md, pdfutils.md, htmlapps.md) to teach ContentNode(uri=...) as the canonical construction pattern, demote VideoNode/AudioNode/DocumentNode/ HTML5AppNode to a legacy/advanced section, and keep exercises and video subtitles (SubtitleFile) as the two documented exceptions. Fixes the nonexistent HTML5ZipFile reference and re-executes the affected notebooks. Co-Authored-By: Claude Sonnet 5 --- docs/concepts/introduction.md | 5 + docs/examples/document_conversion.ipynb | 4 +- docs/examples/exercises.ipynb | 181 +++++- docs/examples/languages.ipynb | 773 +++++++++++++++++++++--- docs/files.md | 108 ++-- docs/htmlapps.md | 8 +- docs/nodes.md | 95 ++- docs/pdfutils.md | 12 +- docs/tutorial/explanations.md | 74 ++- docs/tutorial/gettingstarted.rst | 9 +- docs/tutorial/tutorial.rst | 19 +- examples/gettingstarted/sushichef.py | 12 +- examples/tutorial/sushichef.py | 46 +- 13 files changed, 1033 insertions(+), 313 deletions(-) diff --git a/docs/concepts/introduction.md b/docs/concepts/introduction.md index bd6779dc..975426dc 100644 --- a/docs/concepts/introduction.md +++ b/docs/concepts/introduction.md @@ -51,6 +51,11 @@ Kolibri channels are tree-like structures that consist of the following types of - Slideshow (a sequence of `jpg` and `png` slide images) - Exercises containing questions like multiple choice, multiple selection, and numeric inputs +Chef scripts create each content item by passing a `uri` (a local path or URL) to +`ContentNode` — the file pipeline downloads the resource and infers which of the +kinds above it is from the file extension, so you rarely need to specify the kind +explicitly. + Further reading --------------- diff --git a/docs/examples/document_conversion.ipynb b/docs/examples/document_conversion.ipynb index 13aea321..aa794331 100644 --- a/docs/examples/document_conversion.ipynb +++ b/docs/examples/document_conversion.ipynb @@ -73,7 +73,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "You should now be able to see the file `document.pdf` in the current directory, and add this PDF file to a Kolibri channel using the `DocumentFile` class as part of a `DocumentNode`." + "You should now be able to see the file `document.pdf` in the current directory, and add this PDF file to a Kolibri channel using `ContentNode(uri=...)`." ] }, { @@ -116,7 +116,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "You should now be able to see the file `infographic.pdf` in the current directory, and add this PDF file to a Kolibri channel using the `DocumentFile` class as part of a `DocumentNode`." + "You should now be able to see the file `infographic.pdf` in the current directory, and add this PDF file to a Kolibri channel using `ContentNode(uri=...)`." ] }, { diff --git a/docs/examples/exercises.ipynb b/docs/examples/exercises.ipynb index 1c064878..21f2f7c9 100644 --- a/docs/examples/exercises.ipynb +++ b/docs/examples/exercises.ipynb @@ -25,8 +25,23 @@ { "cell_type": "code", "execution_count": 1, - "metadata": {}, - "outputs": [], + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:53:16.396675Z", + "iopub.status.busy": "2026-07-05T01:53:16.396370Z", + "iopub.status.idle": "2026-07-05T01:53:16.708201Z", + "shell.execute_reply": "2026-07-05T01:53:16.706857Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Unable to load pyppeteer, using phantomjs for JS loading.\n" + ] + } + ], "source": [ "from ricecooker.chefs import SushiChef\n", "from ricecooker.classes.nodes import TopicNode, ExerciseNode\n", @@ -40,7 +55,7 @@ "class ExercisesChef(SushiChef):\n", " channel_info = {\n", " 'CHANNEL_TITLE': 'Sample Exercises',\n", - " 'CHANNEL_SOURCE_DOMAIN': '', # where you got the content\n", + " 'CHANNEL_SOURCE_DOMAIN': 'ricecooker-docs-example.org', # where you got the content\n", " 'CHANNEL_SOURCE_ID': '', # channel's unique id CHANGE ME\n", " 'CHANNEL_LANGUAGE': 'en', # le_utils language code\n", " 'CHANNEL_DESCRIPTION': 'A test channel with different types of exercise questions', # (optional)\n", @@ -92,7 +107,7 @@ " topic.add_child(exercise_node)\n", "\n", " # LOAD JSON DATA (as string) FOR PERSEUS QUESTIONS \n", - " RAW_PERSEUS_JSON_STR = open('../../examples/exercises/chefdata/perseus_graph_question.json', 'r').read()\n", + " RAW_PERSEUS_JSON_STR = open('../../examples/oldexamples/data/perseus_graph_question.json', 'r').read()\n", " # or\n", " # import requests\n", " # RAW_PERSEUS_JSON_STR = requests.get('https://raw.githubusercontent.com/learningequality/sample-channels/master/contentnodes/exercise/perseus_graph_question.json').text\n", @@ -113,6 +128,7 @@ " PerseusQuestion(\n", " id='ex2bQ4',\n", " raw_data=RAW_PERSEUS_JSON_STR,\n", + " ka_language='en',\n", " source_url='https://github.com/learningequality/sample-channels/blob/master/contentnodes/exercise/perseus_graph_question.json'\n", " ),\n", " ]\n", @@ -141,35 +157,150 @@ { "cell_type": "code", "execution_count": 2, - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:53:16.770646Z", + "iopub.status.busy": "2026-07-05T01:53:16.770163Z", + "iopub.status.idle": "2026-07-05T01:53:16.829779Z", + "shell.execute_reply": "2026-07-05T01:53:16.828264Z" + } + }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "\u001b[32mINFO \u001b[0m \u001b[34mIn SushiChef.run method. args={'command': 'dryrun', 'reset': True, 'verbose': True, 'token': 'YOURTO...'} options={}\u001b[0m\n", + "\u001b[32mINFO \u001b[0m \u001b[34mIn SushiChef.run method. args={'command': 'dryrun', 'verbose': True, 'token': 'YOURTO...'} options={}\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ "\u001b[32mINFO \u001b[0m \u001b[34m\n", "\n", "***** Starting channel build process *****\n", "\n", - "\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mCalling construct_channel... \u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Setting up initial channel structure... \u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Validating channel structure...\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Sample Exercises (ChannelNode): 3 descendants\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Math Exercises (TopicNode): 2 descendants\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Basic questions (ExerciseNode): 3 questions\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m An exercise containing a perseus question (ExerciseNode): 1 question\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Tree is valid\n", - "\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mDownloading files...\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mProcessing content...\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m\t*** Processing images for exercise: Basic questions\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m\t*** Images for Basic questions have been processed\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m\t*** Processing images for exercise: An exercise containing a perseus question\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m\t*** Images for An exercise containing a perseus question have been processed\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m All files were successfully downloaded\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mCommand is dryrun so we are not uploading chanel.\u001b[0m\n" + "\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mCalling construct_channel... \u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Setting up initial channel structure... \u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Validating channel structure...\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Sample Exercises (ChannelNode) (): 3 descendants\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Math Exercises (TopicNode) (folder-id): 2 descendants\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Basic questions (ExerciseNode): 3 questions\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m An exercise containing a perseus question (ExerciseNode): 1 question\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Tree is valid\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mDownloading files...\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mProcessing content...\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\t*** Processing images for exercise: Basic questions\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\t*** Images for Basic questions have been processed\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\t*** Processing images for exercise: An exercise containing a perseus question\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\t*** Images for An exercise containing a perseus question have been processed\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m All files were successfully downloaded\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mCommand is dryrun so we are not uploading channel.\u001b[0m\n" ] } ], @@ -227,7 +358,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.4" + "version": "3.13.12" } }, "nbformat": 4, diff --git a/docs/examples/languages.ipynb b/docs/examples/languages.ipynb index 87421145..1b20031a 100644 --- a/docs/examples/languages.ipynb +++ b/docs/examples/languages.ipynb @@ -21,12 +21,19 @@ { "cell_type": "code", "execution_count": 1, - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:51:57.872580Z", + "iopub.status.busy": "2026-07-05T01:51:57.872351Z", + "iopub.status.idle": "2026-07-05T01:51:57.884937Z", + "shell.execute_reply": "2026-07-05T01:51:57.884335Z" + } + }, "outputs": [ { "data": { "text/plain": [ - "Language(native_name='English', primary_code='en', subcode=None, name='English', ka_name=None)" + "Language(native_name='English', primary_code='en', subcode=None, name='English', text_direction='ltr')" ] }, "execution_count": 1, @@ -46,12 +53,19 @@ { "cell_type": "code", "execution_count": 2, - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:51:57.944524Z", + "iopub.status.busy": "2026-07-05T01:51:57.944178Z", + "iopub.status.idle": "2026-07-05T01:51:57.949787Z", + "shell.execute_reply": "2026-07-05T01:51:57.949043Z" + } + }, "outputs": [ { "data": { "text/plain": [ - "Language(native_name='English', primary_code='en', subcode=None, name='English', ka_name=None)" + "Language(native_name='English', primary_code='en', subcode=None, name='English', text_direction='ltr')" ] }, "execution_count": 2, @@ -68,7 +82,14 @@ { "cell_type": "code", "execution_count": 3, - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:51:57.952383Z", + "iopub.status.busy": "2026-07-05T01:51:57.952138Z", + "iopub.status.idle": "2026-07-05T01:51:57.957222Z", + "shell.execute_reply": "2026-07-05T01:51:57.956547Z" + } + }, "outputs": [ { "data": { @@ -89,13 +110,20 @@ { "cell_type": "code", "execution_count": 4, - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:51:57.959601Z", + "iopub.status.busy": "2026-07-05T01:51:57.959370Z", + "iopub.status.idle": "2026-07-05T01:51:57.963559Z", + "shell.execute_reply": "2026-07-05T01:51:57.962575Z" + } + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Language(native_name='Français, langue française', primary_code='fr', subcode=None, name='French', ka_name='francais')\n", + "Language(native_name='Français, langue française', primary_code='fr', subcode=None, name='French', text_direction='ltr')\n", "fr\n" ] } @@ -135,13 +163,27 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], + "execution_count": 5, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:51:57.965548Z", + "iopub.status.busy": "2026-07-05T01:51:57.965320Z", + "iopub.status.idle": "2026-07-05T01:51:58.269810Z", + "shell.execute_reply": "2026-07-05T01:51:58.268950Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Unable to load pyppeteer, using phantomjs for JS loading.\n" + ] + } + ], "source": [ "from ricecooker.chefs import SushiChef\n", - "from ricecooker.classes.nodes import ChannelNode, TopicNode, DocumentNode\n", - "from ricecooker.classes.files import DocumentFile\n", + "from ricecooker.classes.nodes import ChannelNode, TopicNode, ContentNode\n", "from le_utils.constants import licenses\n", "\n", "from le_utils.constants.languages import getlang\n", @@ -154,7 +196,7 @@ " \"\"\"\n", " channel_info = {\n", " 'CHANNEL_TITLE': 'Languages test channel',\n", - " 'CHANNEL_SOURCE_DOMAIN': '', # where you got the content\n", + " 'CHANNEL_SOURCE_DOMAIN': 'ricecooker-docs-example.org', # where you got the content\n", " 'CHANNEL_SOURCE_ID': '', # channel's unique id CHANGE ME!!\n", " 'CHANNEL_LANGUAGE': getlang('mul').code, # set global language for channel\n", " 'CHANNEL_DESCRIPTION': 'This channel contains nodes in multiple languages',\n", @@ -165,51 +207,51 @@ " # create channel\n", " channel = self.get_channel(**kwargs)\n", "\n", - " # create the English topic, add a DocumentNode to it\n", + " # create the English topic, add a ContentNode to it\n", " topic = TopicNode(\n", " source_id=\"\",\n", " title=\"New Topic in English\",\n", " language=getlang('en').code,\n", " )\n", - " doc_node = DocumentNode(\n", + " doc_node = ContentNode(\n", " source_id=\"\",\n", " title='Some doc in English',\n", " description='This is a sample document node in English',\n", - " files=[DocumentFile(path='samplefiles/documents/doc_EN.pdf')],\n", + " uri='samplefiles/documents/doc_EN.pdf',\n", " license=licenses.PUBLIC_DOMAIN,\n", " language=getlang('en').code,\n", " )\n", " topic.add_child(doc_node)\n", " channel.add_child(topic)\n", "\n", - " # create the Spanish topic, add a DocumentNode to it\n", + " # create the Spanish topic, add a ContentNode to it\n", " topic = TopicNode(\n", " source_id=\"\",\n", " title=\"Topic in Spanish\",\n", " language=getlang('es-MX').code,\n", " )\n", - " doc_node = DocumentNode(\n", + " doc_node = ContentNode(\n", " source_id=\"\",\n", " title='Some doc in Spanish',\n", " description='This is a sample document node in Spanish',\n", - " files=[DocumentFile(path='samplefiles/documents/doc_ES.pdf')],\n", + " uri='samplefiles/documents/doc_ES.pdf',\n", " license=licenses.PUBLIC_DOMAIN,\n", " language=getlang('es-MX').code,\n", " )\n", " topic.add_child(doc_node)\n", " channel.add_child(topic)\n", "\n", - " # create the French topic, add a DocumentNode to it\n", + " # create the French topic, add a ContentNode to it\n", " topic = TopicNode(\n", " source_id=\"\",\n", " title=\"Topic in French\",\n", " language=languages.getlang('fr').code,\n", " )\n", - " doc_node = DocumentNode(\n", + " doc_node = ContentNode(\n", " source_id=\"\",\n", " title='Some doc in French',\n", " description='This is a sample document node in French',\n", - " files=[DocumentFile(path='samplefiles/documents/doc_FR.pdf')],\n", + " uri='samplefiles/documents/doc_FR.pdf',\n", " license=licenses.PUBLIC_DOMAIN,\n", " language=getlang('fr').code,\n", " )\n", @@ -229,37 +271,143 @@ { "cell_type": "code", "execution_count": 6, - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:51:58.272743Z", + "iopub.status.busy": "2026-07-05T01:51:58.272375Z", + "iopub.status.idle": "2026-07-05T01:51:58.294460Z", + "shell.execute_reply": "2026-07-05T01:51:58.293999Z" + } + }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "\u001b[32mINFO \u001b[0m \u001b[34mIn SushiChef.run method. args={'command': 'dryrun', 'reset': True, 'verbose': True, 'token': 'YOURTO...'} options={}\u001b[0m\n", + "\u001b[32mINFO \u001b[0m \u001b[34mIn SushiChef.run method. args={'command': 'dryrun', 'verbose': True, 'token': 'YOURTO...'} options={}\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ "\u001b[32mINFO \u001b[0m \u001b[34m\n", "\n", "***** Starting channel build process *****\n", "\n", - "\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mCalling construct_channel... \u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Setting up initial channel structure... \u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Validating channel structure...\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Languages test channel (ChannelNode): 6 descendants\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m New Topic in English (TopicNode): 1 descendant\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Some doc in English (DocumentNode): 1 file\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Topic in Spanish (TopicNode): 1 descendant\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Some doc in Spanish (DocumentNode): 1 file\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Topic in French (TopicNode): 1 descendant\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Some doc in French (DocumentNode): 1 file\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Tree is valid\n", - "\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mDownloading files...\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mProcessing content...\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m\t--- Downloaded e8b1fe37ce3da500241b4af4e018a2d7.pdf\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m\t--- Downloaded cef22cce0e1d3ba08861fc97476b8ccf.pdf\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m\t--- Downloaded 6c8730e3e2554e6eac0ad79304bbcc68.pdf\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m All files were successfully downloaded\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mCommand is dryrun so we are not uploading chanel.\u001b[0m\n" + "\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mCalling construct_channel... \u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Setting up initial channel structure... \u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Validating channel structure...\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Languages test channel (ChannelNode) (): 6 descendants\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m New Topic in English (TopicNode) (): 1 descendant\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Some doc in English (ContentNode): uri: samplefiles/documents/doc_EN.pdf\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Topic in Spanish (TopicNode) (): 1 descendant\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Some doc in Spanish (ContentNode): uri: samplefiles/documents/doc_ES.pdf\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Topic in French (TopicNode) (): 1 descendant\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Some doc in French (ContentNode): uri: samplefiles/documents/doc_FR.pdf\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Tree is valid\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mDownloading files...\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mProcessing content...\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m All files were successfully downloaded\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mCommand is dryrun so we are not uploading channel.\u001b[0m\n" ] } ], @@ -300,20 +448,53 @@ { "cell_type": "code", "execution_count": 7, - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:51:58.297516Z", + "iopub.status.busy": "2026-07-05T01:51:58.297399Z", + "iopub.status.idle": "2026-07-05T01:51:59.540685Z", + "shell.execute_reply": "2026-07-05T01:51:59.540283Z" + } + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "[youtube] FN12ty5ztAs: Downloading webpage\n", - "[youtube] FN12ty5ztAs: Downloading MPD manifest\n", - "dict_keys(['en', 'fr', 'zu'])\n" + "[youtube] Extracting URL: FN12ty5ztAs\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[youtube] FN12ty5ztAs: Downloading webpage\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[youtube] FN12ty5ztAs: Downloading android vr player API JSON\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[info] FN12ty5ztAs: Downloading subtitles: en-5IebwaT_cAk, fr-5IebwaT_cAk, zu-5IebwaT_cAk\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "dict_keys(['en-5IebwaT_cAk', 'fr-5IebwaT_cAk', 'zu-5IebwaT_cAk'])\n" ] } ], "source": [ - "import youtube_dl\n", + "import yt_dlp as youtube_dl\n", "\n", "ydl = youtube_dl.YoutubeDL({\n", " #'quiet': True,\n", @@ -351,18 +532,25 @@ }, { "cell_type": "code", - "execution_count": 10, - "metadata": {}, + "execution_count": 8, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:51:59.543696Z", + "iopub.status.busy": "2026-07-05T01:51:59.543574Z", + "iopub.status.idle": "2026-07-05T01:51:59.594068Z", + "shell.execute_reply": "2026-07-05T01:51:59.593329Z" + } + }, "outputs": [], "source": [ "from ricecooker.chefs import SushiChef\n", "from ricecooker.classes import licenses\n", - "from ricecooker.classes.nodes import ChannelNode, TopicNode, VideoNode\n", - "from ricecooker.classes.files import YouTubeVideoFile, YouTubeSubtitleFile\n", - "from ricecooker.classes.files import is_youtube_subtitle_file_supported_language\n", + "from ricecooker.classes.nodes import ChannelNode, TopicNode, ContentNode\n", + "from ricecooker.classes.files import YouTubeSubtitleFile\n", + "from ricecooker.utils.youtube import is_youtube_subtitle_file_supported_language\n", "\n", "\n", - "import youtube_dl\n", + "import yt_dlp as youtube_dl\n", "ydl = youtube_dl.YoutubeDL({\n", " 'quiet': True,\n", " 'no_warnings': True,\n", @@ -383,7 +571,7 @@ " A sushi chef that creates a channel with content in EN, FR, and SP.\n", " \"\"\"\n", " channel_info = {\n", - " 'CHANNEL_SOURCE_DOMAIN': '', # where you got the content\n", + " 'CHANNEL_SOURCE_DOMAIN': 'ricecooker-docs-example.org', # where you got the content\n", " 'CHANNEL_SOURCE_ID': '', # channel's unique id CHANGE ME!!\n", " 'CHANNEL_TITLE': 'Youtube subtitles downloading chef',\n", " 'CHANNEL_LANGUAGE': 'en',\n", @@ -401,16 +589,17 @@ " subtitle_languages = info[\"subtitles\"].keys()\n", " print('Found subtitle_languages = ', subtitle_languages)\n", " \n", - " # create video node\n", - " video_node = VideoNode(\n", + " # create video node - the pipeline infers the video kind from the youtube.com URL\n", + " video_node = ContentNode(\n", " source_id=youtube_id,\n", " title='Youtube video',\n", " license=TE_LICENSE,\n", " derive_thumbnail=True,\n", - " files=[YouTubeVideoFile(youtube_id=youtube_id)],\n", + " uri='https://www.youtube.com/watch?v={}'.format(youtube_id),\n", " )\n", "\n", " # add subtitles in whichever languages are available.\n", + " # uri alone can't express subtitles - add them explicitly, language is required\n", " for lang_code in subtitle_languages:\n", " if is_youtube_subtitle_file_supported_language(lang_code):\n", " video_node.add_file(\n", @@ -424,58 +613,468 @@ "\n", " channel.add_child(video_node)\n", "\n", - " return channel\n", - "\n", - " " + " return channel" ] }, { "cell_type": "code", - "execution_count": 11, - "metadata": {}, + "execution_count": 9, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-05T01:51:59.596151Z", + "iopub.status.busy": "2026-07-05T01:51:59.596037Z", + "iopub.status.idle": "2026-07-05T01:52:03.645435Z", + "shell.execute_reply": "2026-07-05T01:52:03.644740Z" + } + }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "\u001b[32mINFO \u001b[0m \u001b[34mIn SushiChef.run method. args={'command': 'dryrun', 'reset': True, 'verbose': True, 'token': 'YOURTO...'} options={}\u001b[0m\n", + "\u001b[32mINFO \u001b[0m \u001b[34mIn SushiChef.run method. args={'command': 'dryrun', 'verbose': True, 'token': 'YOURTO...'} options={}\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ "\u001b[32mINFO \u001b[0m \u001b[34m\n", "\n", "***** Starting channel build process *****\n", "\n", - "\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mCalling construct_channel... \u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Setting up initial channel structure... \u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Validating channel structure...\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Youtube subtitles downloading chef (ChannelNode): 1 descendant\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Youtube video (VideoNode): 4 files\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m Tree is valid\n", - "\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mDownloading files...\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mProcessing content...\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34m\t--- Downloaded (YouTube) a144a6af6977247684d2a3977dc6f841.mp4\u001b[0m\n" + "\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mCalling construct_channel... \u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Setting up initial channel structure... \u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Validating channel structure...\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Youtube subtitles downloading chef (ChannelNode) (): 1 descendant\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Youtube video (ContentNode): 3 files\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m Tree is valid\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mDownloading files...\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mProcessing content...\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tInitiating DOWNLOAD for https://edoc.coe.int/4115/postcard-47-flags.jpg with kwargs {'default_ext': 'png'}\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tInitiating DOWNLOAD for https://www.youtube.com/watch?v=FN12ty5ztAs with kwargs {'yt_dlp_settings': {'format': 'bestvideo[height<=480][ext=mp4]+bestaudio[ext=m4a]/bestvideo[height<=480][ext=webm]+bestaudio[ext=webm]/best[height<=480][ext=mp4]'}}\u001b[0m\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Found subtitle_languages = dict_keys(['en-5IebwaT_cAk', 'fr-5IebwaT_cAk', 'zu-5IebwaT_cAk'])\n", + "[youtube] Extracting URL: https://www.youtube.com/watch?v=FN12ty5ztAs\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[youtube] FN12ty5ztAs: Downloading webpage\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tCompleted DOWNLOAD for https://edoc.coe.int/4115/postcard-47-flags.jpg with kwargs {'default_ext': 'png'} saved to docs/examples/storage/e/c/ec609ff23af81c9ffc2eb23103ee1d6e.jpg\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tInitiating CONVERT for docs/examples/storage/e/c/ec609ff23af81c9ffc2eb23103ee1d6e.jpg\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tCompleted CONVERT for docs/examples/storage/e/c/ec609ff23af81c9ffc2eb23103ee1d6e.jpg saved to docs/examples/storage/e/c/ec609ff23af81c9ffc2eb23103ee1d6e.jpg\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: [youtube] No supported JavaScript runtime could be found. Only deno is enabled by default; to use another runtime add --js-runtimes RUNTIME[:PATH] to your command/config. YouTube extraction without a JS runtime has been deprecated, and some formats may be missing. See https://github.com/yt-dlp/yt-dlp/wiki/EJS for details on installing one\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[youtube] FN12ty5ztAs: Downloading android vr player API JSON\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[info] FN12ty5ztAs: Downloading 1 format(s): 134+140\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[download] Destination: /tmp/bd09424388b8b8b966a8afa144edec7d.f134.mp4\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 0.0% of 3.81MiB at 781.64KiB/s ETA 00:05" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 0.1% of 3.81MiB at 1.32MiB/s ETA 00:02" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 0.2% of 3.81MiB at 2.01MiB/s ETA 00:01" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 0.4% of 3.81MiB at 2.42MiB/s ETA 00:01" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 0.8% of 3.81MiB at 1.33MiB/s ETA 00:02" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 1.6% of 3.81MiB at 1.77MiB/s ETA 00:02" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 3.3% of 3.81MiB at 2.02MiB/s ETA 00:01" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 6.5% of 3.81MiB at 2.70MiB/s ETA 00:01" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 13.1% of 3.81MiB at 3.98MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 26.2% of 3.81MiB at 6.32MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 52.4% of 3.81MiB at 5.88MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 100.0% of 3.81MiB at 7.85MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 100% of 3.81MiB in 00:00:00 at 3.88MiB/s " + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[download] Destination: /tmp/bd09424388b8b8b966a8afa144edec7d.f140.m4a\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "Found subtitle_languages = dict_keys(['en', 'fr', 'zu'])\n" + "\r", + "[download] 0.1% of 941.23KiB at 823.54KiB/s ETA 00:01" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 0.3% of 941.23KiB at 1.18MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 0.7% of 941.23KiB at 1.42MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 1.6% of 941.23KiB at 2.16MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 3.3% of 941.23KiB at 3.57MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 6.7% of 941.23KiB at 2.19MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 13.5% of 941.23KiB at 3.11MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 27.1% of 941.23KiB at 4.09MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 54.3% of 941.23KiB at 5.32MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 100.0% of 941.23KiB at 7.03MiB/s ETA 00:00" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\r", + "[download] 100% of 941.23KiB in 00:00:00 at 1.97MiB/s " + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[Merger] Merging formats into \"/tmp/bd09424388b8b8b966a8afa144edec7d.mp4\"\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Deleting original file /tmp/bd09424388b8b8b966a8afa144edec7d.f140.m4a (pass -k to keep)\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Deleting original file /tmp/bd09424388b8b8b966a8afa144edec7d.f134.mp4 (pass -k to keep)\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tCompleted DOWNLOAD for https://www.youtube.com/watch?v=FN12ty5ztAs with kwargs {'yt_dlp_settings': {'format': 'bestvideo[height<=480][ext=mp4]+bestaudio[ext=m4a]/bestvideo[height<=480][ext=webm]+bestaudio[ext=webm]/best[height<=480][ext=mp4]', 'outtmpl': {'default': '/tmp/bd09424388b8b8b966a8afa144edec7d.mp4', 'chapter': '%(title)s - %(section_number)03d %(section_title)s [%(id)s].%(ext)s'}, 'js_runtimes': {'deno': {}}, 'remote_components': set(), 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}, 'forceprint': {}, 'print_to_file': {}}} saved to docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tInitiating CONVERT for docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4 with kwargs {'ffmpeg_settings': {}}\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tCompleted CONVERT for docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4 with kwargs {'ffmpeg_settings': {}} saved to docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tInitiating EXTRACT_METADATA for docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34mVideo preset from docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4 = low resolution\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m\tCompleted EXTRACT_METADATA for docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4 saved to docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4\u001b[0m\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[32mINFO \u001b[0m \u001b[34m All files were successfully downloaded\u001b[0m\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ - "\u001b[32mINFO \u001b[0m \u001b[34m\t--- Downloaded 5f22a71e53271eb2d2abe013457a625d.jpg\u001b[0m\n", - "\u001b[31mERROR \u001b[0m \u001b[34m 3 file(s) have failed to download\u001b[0m\n", - "\u001b[33mWARNING \u001b[0m \u001b[34m\tVideo FN12ty5ztAs: http://www.youtube.com/watch?v=FN12ty5ztAs \n", - "\t Subtitle with langauge en is not available for http://www.youtube.com/watch?v=FN12ty5ztAs\u001b[0m\n", - "\u001b[33mWARNING \u001b[0m \u001b[34m\tVideo FN12ty5ztAs: http://www.youtube.com/watch?v=FN12ty5ztAs \n", - "\t Subtitle with langauge fr is not available for http://www.youtube.com/watch?v=FN12ty5ztAs\u001b[0m\n", - "\u001b[33mWARNING \u001b[0m \u001b[34m\tVideo FN12ty5ztAs: http://www.youtube.com/watch?v=FN12ty5ztAs \n", - "\t Subtitle with langauge zul is not available for http://www.youtube.com/watch?v=FN12ty5ztAs\u001b[0m\n", - "\u001b[32mINFO \u001b[0m \u001b[34mCommand is dryrun so we are not uploading chanel.\u001b[0m\n" + "\u001b[32mINFO \u001b[0m \u001b[34mCommand is dryrun so we are not uploading channel.\u001b[0m\n" ] } ], @@ -535,7 +1134,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.9" + "version": "3.13.12" } }, "nbformat": 4, diff --git a/docs/files.md b/docs/files.md index 70ca7adb..907b5b33 100644 --- a/docs/files.md +++ b/docs/files.md @@ -6,6 +6,12 @@ we first compute `md5` hash of its contents (say `abcdef000000000000000000000000 then store the file at the path `storage/a/b/abcdef00000000000000000000000000.pdf`. The same storage mechanism is used on Kolibri Studio and Kolibri applications. +In most cases you won't construct file objects directly — passing `uri=` to +`ContentNode` (see [nodes.md](./nodes.md) and [concepts/introduction.md](./concepts/introduction.md)) +builds the appropriate file object automatically. The classes below are used +internally by the pipeline, or directly for the two cases `uri` can't express: +exercises and video subtitles. + File objects ------------ @@ -58,8 +64,48 @@ See [languages][./languages.md] to read more about language codes. -Audio files ------------ +Subtitle files +-------------- +Video subtitles are the one case `uri` alone can't express — after creating a +video with `ContentNode(uri=...)`, attach subtitles with +`add_file(SubtitleFile(path=..., language=...))`; `language` is required. + +Subtitle files can be created using +``` +subs_file = SubtitleFile( + path = "file:///path/to/file.vtt", + language = languages.getlang('en').code, +) +``` +Kolibri uses the `.vtt` subtitle format internally, but the following formats can +be automatically converted: `.srt`, `.ttml`, `.scc`, `.dfxp`, and `.sami`. +The subtitle format is inferred from the file extension of the `path` argument. +Use the `subtitlesformat` keyword argument in cases where the path does not end +on a format extension: +``` +subs_file = SubtitleFile( + path = "http:/srtsubs.org/subs/29323923", + subtitlesformat = 'srt', # specify format because not in URL + language = languages.getlang('en').code, +) +``` + +You can also get subtitles using `YouTubeSubtitleFile` which takes a `youtube_id` +and youtube `language` code (may be different from internal language codes). +Use the helper method `is_youtube_subtitle_file_supported_language` to test if +a given youtube language code is supported by `YouTubeSubtitleFile` and skip the +ones that are not currently supported. Please let the LE content team know when +you run into language codes that are not supported so we can add them. + + + +Legacy / advanced: constructing file objects directly +------------------------------------------------------ +These classes are what the pipeline builds internally from `uri` — construct +them directly only when you need `add_file()` for a case `uri` alone can't +express, or need pipeline-internal details. + +### Audio files Use the `AudioFile(DownloadFile)` class to store `mp3` files. @@ -69,8 +115,7 @@ Use the `AudioFile(DownloadFile)` class to store `mp3` files. ) -Document files --------------- +### Document files Use the `DocumentFile` class to add PDF documents: document_file = DocumentFile( @@ -87,18 +132,17 @@ Use the `EPubFile` class to add ePub documents: -HTML files -------------- -The `HTML5ZipFile` class is a generic zip container for web content like HTML, CSS, -and JavaScript. To be a valid `HTML5ZipFile` file, the file must have a `index.html` +### HTML files +The `HTMLZipFile` class is a generic zip container for web content like HTML, CSS, +and JavaScript. To be a valid `HTMLZipFile` file, the file must have a `index.html` in its root. The file `index.html` will be loaded within a sandboxed iframe when this content item is accessed on Kolibri. Chef authors are responsible for scraping the HTML and all the related JS, CSS, and images required to render the web content, and creating the zip file. -Creating a `HTML5ZipFile` is then done using +Creating a `HTMLZipFile` is then done using - document_file = HTML5ZipFile( + document_file = HTMLZipFile( path='/tmp/interactive_js_simulation.zip', language=getlang('en').code ) @@ -113,8 +157,7 @@ Use the `H5PFile` class to add [H5P](https://h5p.org/) files: -Videos files ------------- +### Videos files The following file classes can be added to the `VideoNode`s: class VideoFile(DownloadFile) @@ -161,46 +204,17 @@ WebVideoFiles and YouTubeVideoFiles can also take in __download_settings__ (dict to determine how the video will be downloaded and __high_resolution__ (boolean) to determine what resolution to download. - -Subtitle files can be created using -``` -subs_file = SubtitleFile( - path = "file:///path/to/file.vtt", - language = languages.getlang('en').code, -) -``` -Kolibri uses the `.vtt` subtitle format internally, but the following formats can -be automatically converted: `.srt`, `.ttml`, `.scc`, `.dfxp`, and `.sami`. -The subtitle format is inferred from the file extension of the `path` argument. -Use the `subtitlesformat` keyword argument in cases where the path does not end -on a format extension: -``` -subs_file = SubtitleFile( - path = "http:/srtsubs.org/subs/29323923", - subtitlesformat = 'srt', # specify format because not in URL - language = languages.getlang('en').code, -) -``` - -You can also get subtitles using `YouTubeSubtitleFile` which takes a `youtube_id` -and youtube `language` code (may be different from internal language codes). -Use the helper method `is_youtube_subtitle_file_supported_language` to test if -a given youtube language code is supported by `YouTubeSubtitleFile` and skip the -ones that are not currently supported. Please let the LE content team know when -you run into language codes that are not supported so we can add them. - +See [Subtitle files](#subtitle-files) above for `SubtitleFile`/`YouTubeSubtitleFile`. -Thumbnail files ---------------- +### Thumbnail files The class `ThumbnailFile` defined thumbnails that can be added to channel, topic nodes, and content nodes. The extensions `.png`, `.jpg`, and `.jpeg` and supported. The recommended size for thumbnail images is 400px by 225px (aspect ratio 16:9). -SlideImageFile files --------------------- +### SlideImageFile files The `SlideImageFile` class is used in conjunction with the `SlideshowNode` class to create powerpoint-like slideshow presentations. @@ -231,11 +245,7 @@ Below are some general guidelines for handling video files: - Use max height of `480` for videos that work well in low resolution (most videos) - Use max height of `720` for high resolution videos (lectures with writing on board) - Ricecooker can handle the video compression for you if you specify the - `--compress` command line argument, or by setting the `ffmpeg_settings` property - when creating `VideoFile`s. The default values for `ffmpeg_settings` are as follows: - ``` - ffmpeg_settings = {'crf':32, 'max_height':480 } - ``` + `--compress` command line argument. - The `ffmpeg` setting `crf` stands for Constant Rate Factor and is very useful for controlling overall video quality. Setting `crf=24` produces high quality video (and possibly large file size), `crf=28` is a mid-range quality, and diff --git a/docs/htmlapps.md b/docs/htmlapps.md index cbaff9b9..fe7268f5 100644 --- a/docs/htmlapps.md +++ b/docs/htmlapps.md @@ -221,8 +221,9 @@ with open(indexhtmlpath, 'w') as indexfile: # 3. Zip it! (see https://youtu.be/BODSCrj9FHQ for a laugh) zippath = create_predictable_zip(webroot) ``` -You can then use this zippath as follows `zipfile = HTMLZipFile(path=zippath, ...)` -and add the `zipfile` to a `HTML5AppNode` object using its `add_file` method. +You can then use this zippath directly as the `uri` for a `ContentNode`: +`ContentNode(..., uri=zippath)` — the pipeline recognizes the `.zip` extension +as an HTML5 app. See [here](https://github.com/learningequality/sushi-chef-hplife/blob/550597c211dcaa325a5265c99dc7fbfc71d0b321/transform.py#L43-L72) for a full code sample. @@ -267,8 +268,7 @@ To check if a file exists in the zipfile, use the `contains` method: zipper.contains('index.html') # Returns True zipper.contains('css/style.css') # Returns False ``` -You can then call `zipfile = HTMLZipFile(path=''./myzipfile.zip', ...)` and add -the `zipfile` to a `HTML5AppNode` object using its `add_file` method. +You can then pass `'./myzipfile.zip'` directly as the `uri` for a `ContentNode`. See the source code for more details: [ricecooker/utils/html_writer.py](https://github.com/learningequality/ricecooker/blob/master/ricecooker/utils/html_writer.py). diff --git a/docs/nodes.md b/docs/nodes.md index c1b73b08..b802cf2f 100644 --- a/docs/nodes.md +++ b/docs/nodes.md @@ -16,24 +16,18 @@ questions object (defined in `ricecooker.classes.questions`) which are created s Overview -------- -The following diagram lists all the node classes defined in `ricecooker.classes.nodes` -and shows the associated file and question classes that content nodes can contain. +The following diagram shows the node classes defined in `ricecooker.classes.nodes`. +Content is added using `ContentNode(uri=...)` — see [concepts/introduction.md](./concepts/introduction.md) +for how the pipeline infers kind, preset, and files from `uri`. `ExerciseNode` is +the other primary leaf class, since exercise questions can't be expressed as a `uri`. ricecooker.classes.nodes | - | ricecooker.classes.files - class Node(object) | - class ChannelNode(Node) | - class TreeNode(Node) | - class TopicNode(TreeNode) | - class ContentNode(TreeNode) | - class AudioNode(ContentNode) files = [AudioFile] - class DocumentNode(ContentNode) files = [DocumentFile, EPubFile] - class HTML5AppNode(ContentNode) files = [HTMLZipFile] - class H5PAppNode(ContentNode) files = [H5PFile] - class SlideshowNode(ContentNode) files = [SlideImageFile] - class VideoNode(ContentNode) files = [VideoFile, WebVideoFile, YouTubeVideoFile, - SubtitleFile, YouTubeSubtitleFile] + class Node(object) + class ChannelNode(Node) + class TreeNode(Node) + class TopicNode(TreeNode) + class ContentNode(TreeNode) uri='...' --> pipeline infers kind, preset, and files class ExerciseNode(ContentNode) questions = [SingleSelectQuestion, MultipleSelectQuestion, InputQuestion, @@ -42,6 +36,11 @@ and shows the associated file and question classes that content nodes can contai | ricecooker.classes.questions +See [Legacy / advanced node classes](#legacy-advanced-node-classes) below for +the `VideoNode`/`AudioNode`/`DocumentNode`/`HTML5AppNode`/`H5PAppNode` subclasses, +and [SlideshowNode nodes](#slideshownode-nodes) for `SlideshowNode`, which has +no `uri`-based equivalent. + In the remainder of this document we'll describe in full detail the metadata that is needed to specify different content nodes. @@ -174,45 +173,21 @@ the thumbnails of the content nodes they contain. Content nodes ------------- -The table summarizes summarizes the content node classes, their associated files, -and the file formats supported by each file class: - - ricecooker.classes.nodes ricecooker.classes.files - | | - AudioNode --files--> AudioFile # .mp3 - DocumentNode --files--> DocumentFile # .pdf - EPubFile # .epub - SlideshowNode --files--> SlideImageFile # .png/.jpg - HTML5AppNode --files--> HTMLZipFile # .zip - VideoNode --files--> VideoFile, WebVideoFile, YouTubeVideoFile, # .mp4 - SubtitleFile, YouTubeSubtitleFile # .vtt - - -For your copy-paste convenience, here is the sample code for creating a content -node (`DocumentNode`) and an associated (`DocumentFile`) - - content_node = DocumentNode( - source_id='', - title='Some Document', - author='First Last (author\'s name)', - description='Put node description here', - language=getlang('en').code, - license=get_license(licenses.CC_BY, copyright_holder='Copyright holder name'), - thumbnail='some/local/path/name_thumb.jpg', - files=[DocumentFile( - path='some/local/path/name.pdf', - language=getlang('en').code - )] +For your copy-paste convenience, here is the sample code for creating a content node: + + content_node = ContentNode( + source_id='pubs/mafri-potatoe', + title='Growing potatoes', + description='An article about growing potatoes on your rooftop.', + license=get_license('CC BY', copyright_holder='University of Alberta'), + language='en', + uri='https://www.gov.mb.ca/inr/pdf/pubs/mafri-potatoe.pdf', ) -Files can be passed in upon initialization as in the above sample, or can be -added after initialization using the content_node's `add_files` method. - -Note you also use URLs for `path` and `thumbnail` instead of local filesystem paths, -and the files will be downloaded for you automatically. - -You can replace `DocumentNode` and `DocumentFile` with any of the other combinations -of content node and file types. +The pipeline infers the file(s), format preset, and content `kind` from `uri` — +see [files.md](./files.md) and [concepts/introduction.md](./concepts/introduction.md#kolibri-channels). +`uri` accepts a URL or a local filesystem path, and the resource is downloaded +for you automatically. Specify `derive_thumbnail=True` and leave thumbnail blank (`thumbnail=None`) to let Ricecooker automatically generate a thumbnail for the node based on its content. @@ -325,6 +300,22 @@ for which you can also a [rendered preview here](http://khan.github.io/perseus/? +Legacy / advanced node classes +------------------------------- +**Deprecated:** `VideoNode`, `AudioNode`, `DocumentNode`, `HTML5AppNode`, and `H5PAppNode` +are legacy backwards-compatibility classes. New chef scripts should use +`ContentNode(uri=...)` instead — the pipeline infers everything these +subclasses used to require you to specify by hand. These classes remain +supported and are not scheduled for removal, but are no longer the +documented happy path. + +They remain useful when you need to attach files the pipeline can't infer +from a single `uri` alone — e.g. a `VideoNode` with an attached `SubtitleFile` +(see [files.md](./files.md#subtitle-files)) — though the same effect is +achieved more simply with `ContentNode(uri=...)` plus `add_file(SubtitleFile(...))`, +as shown in the [tutorial](../tutorial/tutorial.html#step-6-add-content). + + SlideshowNode nodes ------------------- The `SlideshowNode` class and the associated `SlideImageFile` class are used to diff --git a/docs/pdfutils.md b/docs/pdfutils.md index 5ab73da3..41373bde 100644 --- a/docs/pdfutils.md +++ b/docs/pdfutils.md @@ -35,16 +35,16 @@ The output `chapters` is list of dictionaries with `title` and `path` attributes ... ] -Use this information to create an individual `DocumentNode` for each PDF and store +Use this information to create an individual `ContentNode` for each PDF and store them in a `TopicNode` that corresponds to the book: - from ricecooker.classes import nodes, files + from ricecooker.classes.nodes import ContentNode, TopicNode - book_node = nodes.TopicNode(title='Book title', description='Book description') + book_node = TopicNode(title='Book title', description='Book description') for chapter in chapters: - chapter_node = nodes.DocumentNode( + chapter_node = ContentNode( title=chapter['title'], - files=[files.DocumentFile(chapter['path'])], + uri=chapter['path'], ... ) book_node.add_child(chapter_node) @@ -130,7 +130,7 @@ tree of title and paths: ] You'll need to create a `TopicNode` for each chapter that has `children` and -create a `DocumentNode` for each of the children of that chapter. +create a `ContentNode` for each of the children of that chapter. diff --git a/docs/tutorial/explanations.md b/docs/tutorial/explanations.md index aa05d2a9..2c749004 100644 --- a/docs/tutorial/explanations.md +++ b/docs/tutorial/explanations.md @@ -18,8 +18,7 @@ tutorial and walk through and comment on the most important parts of the code. ```python #!/usr/bin/env python from ricecooker.chefs import SushiChef -from ricecooker.classes.nodes import TopicNode, DocumentNode -from ricecooker.classes.files import DocumentFile +from ricecooker.classes.nodes import TopicNode, ContentNode from ricecooker.classes.licenses import get_license class SimpleChef(SushiChef): # (1) @@ -36,30 +35,24 @@ class SimpleChef(SushiChef): # ( channel = self.get_channel(**kwargs) # (3) potato_topic = TopicNode(title="Potatoes!", source_id="patates") # (4) channel.add_child(potato_topic) # (5) - doc_node = DocumentNode( # (6) + doc_node = ContentNode( # (6) title='Growing potatoes', description='An article about growing potatoes on your rooftop.', source_id='inr/pdf/pubs/mafri-potatoe.pdf', - author=None, - language='en', # (7) - license=get_license('CC BY', copyright_holder='U. of Alberta'), # (8) - files=[ - DocumentFile( # (9) - path='https://www.gov.mb.ca/inr/pdf/pubs/mafri-potatoe.pdf', # (10) - language='en', # (11) - ) - ], + license=get_license('CC BY', copyright_holder='U. of Alberta'), # (7) + language='en', # (8) + uri='https://www.gov.mb.ca/inr/pdf/pubs/mafri-potatoe.pdf', # (9) ) potato_topic.add_child(doc_node) return channel -if __name__ == '__main__': # (12) +if __name__ == '__main__': # (10) """ Run this script on the command line using: python simple_chef.py --token=YOURTOKENHERE9139139f3a23232 """ simple_chef = SimpleChef() - simple_chef.main() # (13) + simple_chef.main() # (11) ``` @@ -125,40 +118,41 @@ you to a structure that best first the needs of learners and teachers. ### Content nodes -The `ricecooker` library provides classes like `DocumentNode`, `VideoNode`, -`AudioNode`, etc., to store the metadata associate with content items. -Each content node also has one or more files associated with it, -`EPubFile`, `DocumentFile`, `VideoFile`, `AudioFile`, `ThumbnailFile`, etc. - -Line (6) shows how to create a `DocumentNode` to store the metadata for a pdf file. -The `title` and `description` attributes are set. We also set the `source_id` -attribute to a unique identifier for this document. The document does not specify authors, -so we set the `author` attribute to `None`. - -On (7), we set `language` attribute to the internal language code `en`, to indicate -the content node is in English. We use the same language code later on line (11) -to indicate the file contents are in English. The Python package `le-utils` defines -the internal language codes used throughout the Kolibri platform (e.g. `en`, `es-MX`, and `zul`). -To find the internal language code for a given language, you can locate it in the -[lookup table](https://github.com/learningequality/le-utils/blob/master/le_utils/resources/languagelookup.json), -or use one of the language lookup helper functions defined in `le_utils.constants.languages`. +The canonical way to add a content item is `ContentNode(source_id, title, license, uri=...)`. +Passing a local path or URL as `uri` hands it to the file pipeline (transfer → convert → +extract_metadata), which infers the content's kind and format preset and builds the +underlying file object automatically — you don't need to construct a file object yourself +for the vast majority of content. + +Line (6) shows how to create a `ContentNode` to store the metadata for a pdf file. +The `title` and `description` attributes are set, and `source_id` is set to a unique +identifier for this document. -Line (8) shows how we set the `license` attribute to the appropriate instance of +Line (7) shows how we set the `license` attribute to the appropriate instance of `ricecooker.classes.licenses.License`. All non-topic nodes must be assigned a license upon initialization. You can obtain the appropriate license object using the helper function `get_license` defined in `ricecooker.classes.licenses`. Use the predefined license ids given in `le_utils.constants.licenses` as the first argument to the `get_license` helper function. +On (8), we set the `language` attribute to the internal language code `en`, to indicate +the content node is in English. The Python package `le-utils` defines the internal +language codes used throughout the Kolibri platform (e.g. `en`, `es-MX`, and `zul`). +To find the internal language code for a given language, you can locate it in the +[lookup table](https://github.com/learningequality/le-utils/blob/master/le_utils/resources/languagelookup.json), +or use one of the language lookup helper functions defined in `le_utils.constants.languages`. -### Files -On lines (9, 10, and 11), we create a `DocumentFile` instance and set the appropriate -`path` and `language` attributes. Note that `path` can be either a local filesystem path, -or a web URL (as in the above example). Paths that point to web URLs will be -downloaded automatically when the chef runs and cached locally. Note the default -ricecooker behaviour is to cache downloaded files forever. -Use the `--update` argument to bypass the cached and re-download all files. -The `--update` must be used whenever files are modified but the path stays the same. +Line (9) shows the `uri` attribute — the path or URL to the actual pdf file. `uri` can be +either a local filesystem path or a web URL (as in the above example); URLs are downloaded +automatically when the chef runs and cached locally. Note the default ricecooker behaviour +is to cache downloaded files forever. Use the `--update` argument to bypass the cache and +re-download all files. The `--update` must be used whenever files are modified but the path +stays the same. + +The only cases `uri` alone can't express are exercises (`ExerciseNode` + questions) and +video subtitles (`SubtitleFile`, which requires an explicit `language`) — see +[Legacy / advanced node classes](../nodes.md#legacy-advanced-node-classes) and +[Subtitle files](../files.md#subtitle-files). diff --git a/docs/tutorial/gettingstarted.rst b/docs/tutorial/gettingstarted.rst index 23eb3ec1..2f7fd512 100644 --- a/docs/tutorial/gettingstarted.rst +++ b/docs/tutorial/gettingstarted.rst @@ -37,15 +37,12 @@ and pass in the command line argument ``--token=path/to/studiotoken.txt``. Video overview -------------- -Watch this `video tutorial `__ -to learn how to create a new content integration script and set the required +Watch this video tutorial to learn how to create a new content integration script and set the required channel metadata fields like ``CHANNEL_SOURCE_DOMAIN`` and ``CHANNEL_SOURCE_ID``. .. raw:: html - - - +
 
@@ -110,7 +107,7 @@ If the command succeeds, you should see something like this printed in your term In SushiChef.run method. args={'command': 'uploadchannel', 'token': '', 'update': False, 'resume': False, 'stage': True, 'publish': False} options={} Logged in with username you@yourdomain.org - Ricecooker v0.6.42 is up-to-date. + Ricecooker vX.Y.Z is up-to-date. ***** Starting channel build process ***** diff --git a/docs/tutorial/tutorial.rst b/docs/tutorial/tutorial.rst index b1fb8e2a..bcc8c377 100644 --- a/docs/tutorial/tutorial.rst +++ b/docs/tutorial/tutorial.rst @@ -89,14 +89,17 @@ Step 5: Create a Subtopic | | Your Subtopic -Step 6: Create Files --------------------- -1. Go to the next **TODO** in the sushichef.py file. Here, you will create a pdf file -2. Copy/paste the example code and change ``examplepdf`` to ``mypdf``. - ``DocumentFile(...)`` will automatically download a pdf file from the given path. -3. Set the ``source_id``, the ``title``, and the ``path`` (any url to a pdf file) -4. Repeat steps 1-3 for video files and audio files. -5. Finally, add your files to your channel (see last \*\* statements) +Step 6: Add Content +------------------- +1. Go to the next **TODO** in the sushichef.py file. Here, you will create a pdf content node +2. Copy/paste the example ``ContentNode(...)`` block and change ``examplepdf`` to ``mypdf``. +3. Set the ``source_id``, the ``title``, and the ``uri`` (any URL to a ``.pdf`` file) — + the pipeline downloads the file and infers it's a document automatically. +4. Repeat steps 1-3 for the video and audio content nodes. +5. Finally, add your content nodes to your channel (see last \*\* statements) + +Note the video example also attaches a subtitle file — see the +`explanations page `__ for why ``uri`` alone can't express that case. :: diff --git a/examples/gettingstarted/sushichef.py b/examples/gettingstarted/sushichef.py index 5029ec5d..0161dafb 100644 --- a/examples/gettingstarted/sushichef.py +++ b/examples/gettingstarted/sushichef.py @@ -1,8 +1,7 @@ #!/usr/bin/env python from ricecooker.chefs import SushiChef -from ricecooker.classes.files import DocumentFile from ricecooker.classes.licenses import get_license -from ricecooker.classes.nodes import DocumentNode +from ricecooker.classes.nodes import ContentNode from ricecooker.classes.nodes import TopicNode @@ -20,18 +19,13 @@ def construct_channel(self, **kwargs): channel = self.get_channel(**kwargs) potato_topic = TopicNode(title="Potatoes!", source_id="") channel.add_child(potato_topic) - document_node = DocumentNode( + document_node = ContentNode( title="Growing potatoes", description="An article about growing potatoes on your rooftop.", source_id="pubs/mafri-potatoe", license=get_license("CC BY", copyright_holder="University of Alberta"), language="en", - files=[ - DocumentFile( - path="https://www.gov.mb.ca/inr/pdf/pubs/mafri-potatoe.pdf", - language="en", - ) - ], + uri="https://www.gov.mb.ca/inr/pdf/pubs/mafri-potatoe.pdf", ) potato_topic.add_child(document_node) return channel diff --git a/examples/tutorial/sushichef.py b/examples/tutorial/sushichef.py index cc6733b2..692d1353 100644 --- a/examples/tutorial/sushichef.py +++ b/examples/tutorial/sushichef.py @@ -2,14 +2,10 @@ from le_utils.constants import licenses from ricecooker.chefs import SushiChef -from ricecooker.classes.files import AudioFile -from ricecooker.classes.files import DocumentFile -from ricecooker.classes.files import VideoFile +from ricecooker.classes.files import SubtitleFile from ricecooker.classes.licenses import get_license -from ricecooker.classes.nodes import AudioNode -from ricecooker.classes.nodes import DocumentNode +from ricecooker.classes.nodes import ContentNode from ricecooker.classes.nodes import TopicNode -from ricecooker.classes.nodes import VideoNode class TutorialChef(SushiChef): @@ -61,44 +57,44 @@ def construct_channel(self, *args, **kwargs): # Content # You can add documents (pdfs and ePubs), videos, audios, and other content ######################################################################## - # let's create a document file called 'Example PDF' - document_file = DocumentFile(path="http://www.pdf995.com/samples/pdf.pdf") - examplepdf = DocumentNode( + # let's create a document node called 'Example PDF' + examplepdf = ContentNode( title="Example PDF", source_id="example-pdf", - files=[document_file], license=get_license(licenses.PUBLIC_DOMAIN), + uri="http://www.pdf995.com/samples/pdf.pdf", ) - # TODO: Create your pdf file here (use any url to a .pdf file) + # TODO: Create your pdf node here (use any url to a .pdf file) - # We are also going to add a video file called 'Example Video' - video_file = VideoFile( - path="https://ia600209.us.archive.org/27/items/RiceChef/Rice Chef.mp4" - ) + # We are also going to add a video node called 'Example Video' fancy_license = get_license( licenses.SPECIAL_PERMISSIONS, description="Special license for ricecooker fans only.", copyright_holder="The chef video makers", ) - examplevideo = VideoNode( + examplevideo = ContentNode( title="Example Video", source_id="example-video", - files=[video_file], license=fancy_license, + uri="https://ia600209.us.archive.org/27/items/RiceChef/Rice Chef.mp4", ) - # TODO: Create your video file here (use any url to a .mp4 file) - - # Finally, we are creating an audio file called 'Example Audio' - audio_file = AudioFile( - path="https://ia802508.us.archive.org/5/items/testmp3testfile/mpthreetest.mp3" + # uri alone can't express subtitles - add them explicitly, language is required + examplevideo.add_file( + SubtitleFile( + path="https://raw.githubusercontent.com/learningequality/ricecooker/main/examples/oldexamples/content/captions.vtt", + language="sw", + ) ) - exampleaudio = AudioNode( + # TODO: Create your video node here (use any url to a .mp4 file) + + # Finally, we are creating an audio node called 'Example Audio' + exampleaudio = ContentNode( title="Example Audio", source_id="example-audio", - files=[audio_file], license=get_license(licenses.PUBLIC_DOMAIN), + uri="https://ia802508.us.archive.org/5/items/testmp3testfile/mpthreetest.mp3", ) - # TODO: Create your audio file here (use any url to a .mp3 file) + # TODO: Create your audio node here (use any url to a .mp3 file) # Now that we have our files, let's add them to our channel channel.add_child(examplepdf) # Adding 'Example PDF' to your channel From 2de59ad5c04cfbb26bc8fff50817372f2c19bde5 Mon Sep 17 00:00:00 2001 From: rtibblesbot Date: Sun, 5 Jul 2026 10:50:33 -0700 Subject: [PATCH 2/5] Address PR #683 review feedback - Fix dead le-utils master-branch links across docs (repo's default branch is main, not master) - Replace the dead archive.org RiceChef tutorial video URL with a live public-domain video - Document getting YouTube subtitles automatically via YouTubeSubtitleFile alongside ContentNode(uri=) - Recommend ContentNode's thumbnail attribute over a standalone ThumbnailFile - Land the previously-missing Phase 4 doc fixes: installation.md uv/pyproject rewrite, chefops.md CLI-flag drift, video_compression.md ffmpeg_settings cleanup, downloader.md import-path fix - Verify the full doc set builds cleanly under Sphinx (no new warnings vs. baseline) Co-Authored-By: Claude Sonnet 5 --- docs/chefops.md | 32 ++++++++++++++++++++-------- docs/concepts/developer_workflows.md | 2 +- docs/csv_metadata/csv_exercises.md | 2 +- docs/downloader.md | 2 +- docs/files.md | 18 +++++++++++++++- docs/installation.md | 32 ++++++++++++++++++---------- docs/languages.md | 4 ++-- docs/nodes.md | 8 +++---- docs/tutorial/explanations.md | 2 +- docs/tutorial/gettingstarted.rst | 2 +- docs/video_compression.md | 3 --- examples/tutorial/sushichef.py | 2 +- 12 files changed, 73 insertions(+), 36 deletions(-) diff --git a/docs/chefops.md b/docs/chefops.md index b25e15f8..3752d738 100644 --- a/docs/chefops.md +++ b/docs/chefops.md @@ -9,12 +9,14 @@ Ricecooker CLI -------------- This listing shows the `ricecooker` command line interface (CLI) arguments: - usage: sushichef.py [-h] [--token TOKEN] [-u] [-v] [--quiet] [--warn] - [--debug] [--compress] [--thumbnails] - [--resume] [--step {CONSTRUCT_CHANNEL, CREATE_TREE, - DOWNLOAD_FILES, GET_FILE_DIFF, - START_UPLOAD, UPLOAD_CHANNEL}] - [--deploy] [--publish] + usage: sushichef.py [-h] [--token TOKEN] [-u] [--debug] [-v] [--warn] + [--quiet] [--compress] [--thumbnails] + [--download-attempts DOWNLOAD_ATTEMPTS] [--resume] + [--step {INIT, CONSTRUCT_CHANNEL, CREATE_TREE, + DOWNLOAD_FILES, GET_FILE_DIFF, START_UPLOAD, + UPLOADING_FILES, UPLOAD_CHANNEL, PUBLISH_CHANNEL, + DONE, LAST}] + [--prompt] [--deploy] [--publish] [--sample SIZE] required arguments: --token TOKEN Studio API Access Token (specify wither the token @@ -22,18 +24,30 @@ This listing shows the `ricecooker` command line interface (CLI) arguments: optional arguments: -h, --help show this help message and exit + -u, --update Force file re-download (skip .ricecookerfilecache/). --debug Print extra debugging infomation. -v, --verbose Verbose mode (default). + --warn Print errors and warnings. + --quiet Print only errors. --compress Compress videos using ffmpeg -crf=32 -b:a 32k mono. --thumbnails Automatically generate thumbnails for content nodes. + --download-attempts N Maximum number of times to retry downloading files (default: 3). --resume Resume chef session from a specified step. - --step {INIT, ... Step to resume progress from (must be used with --resume flag) - --update Force re-download of files (skip .ricecookerfilecache/ check) - --sample SIZE Upload a sample of SIZE nodes from the channel. + --step {INIT, CONSTRUCT_CHANNEL, CREATE_TREE, DOWNLOAD_FILES, + GET_FILE_DIFF, START_UPLOAD, UPLOADING_FILES, UPLOAD_CHANNEL, + PUBLISH_CHANNEL, DONE, LAST} + Step to resume progress from (use with --resume). + --prompt Prompt user to open the channel after the chef run. --deploy Immediately deploy changes to channel's main tree. This operation will overwrite the previous channel content. Use only during development. --publish Publish newly uploaded version of the channel. + --sample SIZE Upload a sample of SIZE nodes from the channel. + +**Staging is the default.** Ricecooker uploads a draft/staging tree by default, +so a chef run never overwrites the channel's live main tree unless you pass +`--deploy`. Use `--deploy` only when you intentionally want to overwrite the +main tree. As you can tell, there are lot of arguments to choose from, and this is not even the complete list: you'll have to run `./sushichef.py -h` to see the latest version. diff --git a/docs/concepts/developer_workflows.md b/docs/concepts/developer_workflows.md index 54ab52c5..c727ede1 100644 --- a/docs/concepts/developer_workflows.md +++ b/docs/concepts/developer_workflows.md @@ -129,7 +129,7 @@ Use the following rubric as a checklist to know when a sushi chef script is done ### Metadata checks 1. Do all nodes have appropriate titles? 2. Do all nodes have appropriate descriptions (when available in the source)? -3. Is the correct [language code](https://github.com/learningequality/le-utils/blob/master/le_utils/resources/languagelookup.json) +3. Is the correct [language code](https://github.com/learningequality/le-utils/blob/main/le_utils/resources/languagelookup.json) set on all nodes and files? 4. Is the `license` field set to the correct value for all nodes? 5. Is the `source_id` field set consistently for all content nodes? diff --git a/docs/csv_metadata/csv_exercises.md b/docs/csv_metadata/csv_exercises.md index 57c261f8..0b9a2046 100644 --- a/docs/csv_metadata/csv_exercises.md +++ b/docs/csv_metadata/csv_exercises.md @@ -51,7 +51,7 @@ Individual questions - `Source ID *`: This field is the link (foreign key) to the an exercise node, e.g. `exrc1` - `Question ID *`: A unique identifier for this question within the exercise, e.g. q1 - `Question type *`: (str) Question types are defined in - [le-utils](https://github.com/learningequality/le-utils/blob/master/le_utils/constants/exercises.py#L34). + [le-utils](https://github.com/learningequality/le-utils/blob/main/le_utils/constants/exercises.py#L34). The currently supported question types for the CSV workflow are: - `input_question`: Numeric input question, e.g. What is 2+2? - `single_selection`: Multiple choice questions where a single correct answer. diff --git a/docs/downloader.md b/docs/downloader.md index 749b5b8a..12379eba 100644 --- a/docs/downloader.md +++ b/docs/downloader.md @@ -32,7 +32,7 @@ pages that correctly include all necessary resources without extra code. * If you need to make modifications, call `create_zip_dir_for_page`, then modify the files in the directory it returns as needed. (Not modifying the original sources allows you to keep a clean copy at all times.) Finally, create a ZIP by - calling `ricecooker.utils.create_predictable_zip` and use the zip created + calling `ricecooker.utils.zip.create_predictable_zip` and use the zip created as a file for an HTML5 app node. Usage example: diff --git a/docs/files.md b/docs/files.md index 907b5b33..977664b3 100644 --- a/docs/files.md +++ b/docs/files.md @@ -50,7 +50,7 @@ current working directory of the chef script, or the URL of a web resource. ### Language The Python package `le-utils` defines the internal language codes used throughout the Kolibri platform (e.g. `en`, `es-MX`, and `zul`). To find the internal language -code for a given language, you can locate it in the [lookup table](https://github.com/learningequality/le-utils/blob/master/le_utils/resources/languagelookup.json), +code for a given language, you can locate it in the [lookup table](https://github.com/learningequality/le-utils/blob/main/le_utils/resources/languagelookup.json), or use one of the language lookup helper functions defined in `le_utils.constants.languages`: - `getlang() --> lang_obj`: basic lookup used to ensure `` is a valid internal language code (otherwise returns `None`). @@ -97,6 +97,17 @@ a given youtube language code is supported by `YouTubeSubtitleFile` and skip the ones that are not currently supported. Please let the LE content team know when you run into language codes that are not supported so we can add them. +`YouTubeSubtitleFile` pairs with the `ContentNode(uri=...)` approach for YouTube +videos: passing a YouTube URL as `uri` downloads the video, and `YouTubeSubtitleFile` +downloads the caption track directly from YouTube — no local `.vtt` file needed: +``` +video_node = ContentNode( + source_id, title, license, + uri=f"https://www.youtube.com/watch?v={youtube_id}", +) +video_node.add_file(YouTubeSubtitleFile(youtube_id=youtube_id, language="en")) +``` + Legacy / advanced: constructing file objects directly @@ -211,6 +222,11 @@ See [Subtitle files](#subtitle-files) above for `SubtitleFile`/`YouTubeSubtitleF The class `ThumbnailFile` defined thumbnails that can be added to channel, topic nodes, and content nodes. The extensions `.png`, `.jpg`, and `.jpeg` and supported. +In general it's recommended to add a thumbnail using the `thumbnail` attribute on +`ChannelNode`/`TopicNode`/`ContentNode` (e.g. `ContentNode(..., thumbnail="path/or/url/to/image.png")`) +rather than constructing a `ThumbnailFile` and calling `add_file()` — the `thumbnail` +attribute accepts the same local path or URL and wraps it in a `ThumbnailFile` for you. + The recommended size for thumbnail images is 400px by 225px (aspect ratio 16:9). diff --git a/docs/installation.md b/docs/installation.md index a63cea9f..399206b6 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,10 +1,11 @@ Installation ============ -You can install `ricecooker` by running the command `pip install ricecooker`, -which will install the Python package and all its Python dependencies. -You'll need version 3.9 or higher of Python to use the `ricecooker` framework, -as well as some software for media file conversions (`ffmpeg` and `poppler`). +Ricecooker chef scripts are typically developed inside their own project, managed +with [`uv`](https://docs.astral.sh/uv/). Add `ricecooker` as a dependency in your +chef project's `pyproject.toml` and run `uv sync` to install it and its Python +dependencies. You'll need Python 3.9-3.13 (matching `ricecooker`'s supported +range), as well as some software for media file conversions (`ffmpeg` and `poppler`). In the next fifteen minutes or so, we'll setup your computer with all these things so you can get started writing your first content integration scripts. @@ -27,7 +28,7 @@ On a Debian or Ubuntu GNU/Linux, you can install the necessary packages using: You may need to adjust the package names for other Linux distributions (ContOS/Fedora/OpenSuSE). -*Checklist*: verify your python version is 3.9 or higher by running `python3 --version`. +*Checklist*: verify your python version is between 3.9 and 3.13 by running `python3 --version`. If no `python3` command exists, then try `python --version`. Run the commands `ffmpeg -h` and `pdftoppm -h` to make sure they are available. @@ -37,7 +38,7 @@ Mac OS X users can install the necessary software using [Homebrew](https://brew. brew install git python3 ffmpeg poppler -*Checklist*: verify you python version is 3.9 or higher by running `python3 --version`. +*Checklist*: verify your python version is between 3.9 and 3.13 by running `python3 --version`. Also run the commands `ffmpeg -h` and `pdftoppm -h` to make sure they are available. @@ -59,7 +60,7 @@ several programs and make sure their `bin`-directories are added to the `Path` v to your user Path variable following [these instructions](https://www.computerhope.com/issues/ch000549.htm). - *Checklist*: Open a new command prompt and type in `ffmpeg -h` and `ffprobe -h` to verify the commands `ffmpeg` and `ffprobe` are available on your Path. -3. Download the file linked under "Latest binary" from [poppler-windows](http://blog.alivate.com.au/poppler-windows/). +3. Download the latest release archive from [poppler-windows](https://github.com/oschwartz10612/poppler-windows/releases/). You will need to download and install [7-zip](https://www.7-zip.org/) to "unzip" the `.7z` archive. Extract the file to a some permanent location in your files. Add the `bin` folder `poppler-0.xx.y\bin` to your Path variable. @@ -82,12 +83,21 @@ installation steps and also explains the adding-to-Path process. Installing Ricecooker --------------------- -To install the `ricecooker` package, simply run this command in a command prompt: +Create a `pyproject.toml` for your chef project (or use an existing one), then run: - pip install ricecooker + uv add ricecooker -You will see lots of lines scroll on the screen as `pip`, the package installer for Python, -installs all the Python packages required to create content integration scripts. +`uv` resolves and installs `ricecooker` and all the Python packages required to +create content integration scripts into your project's `.venv`, and records the +dependency in `pyproject.toml`. + + +Contributing to ricecooker itself +---------------------------------- +If you're contributing to `ricecooker` itself (not just writing a chef script), +clone the repo and run `uv sync --group dev` to install all dev dependencies, +`uv run --group test pytest` to run the test suite, and `uvx prek run --all-files` +to lint before committing — see `AGENTS.md` for the full contributor quick start. **Reporting issues**: If you run into problems or encounter an error in any of the above steps, please let us know by [opening an issue on github](https://github.com/learningequality/ricecooker/issues). diff --git a/docs/languages.md b/docs/languages.md index d18e4feb..55d97f07 100644 --- a/docs/languages.md +++ b/docs/languages.md @@ -1,8 +1,8 @@ Kolibri Language Codes ---------------------- -The file [le_utils/constants/languages.py](https://github.com/learningequality/le-utils/blob/master/le_utils/constants/languages.py) -and the lookup table in [le_utils/resources/languagelookup.json](https://github.com/learningequality/le-utils/blob/master/le_utils/resources/languagelookup.json) +The file [le_utils/constants/languages.py](https://github.com/learningequality/le-utils/blob/main/le_utils/constants/languages.py) +and the lookup table in [le_utils/resources/languagelookup.json](https://github.com/learningequality/le-utils/blob/main/le_utils/resources/languagelookup.json) define the internal representation for languages codes used by Ricecooker, Kolibri, and Kolibri Studio to identify content items in different languages. diff --git a/docs/nodes.md b/docs/nodes.md index b802cf2f..68b1b405 100644 --- a/docs/nodes.md +++ b/docs/nodes.md @@ -93,7 +93,7 @@ content nodes that aren't for that video need to have different `source_id`s. ### Licenses All content nodes within Kolibri and Kolibri Studio must have a license. The file -[le_utils/constants/licenses.py](https://github.com/learningequality/le-utils/blob/master/le_utils/constants/licenses.py) +[le_utils/constants/licenses.py](https://github.com/learningequality/le-utils/blob/main/le_utils/constants/licenses.py) contains the constants used to identify the license types. These constants are meant to be used in conjunction with the helper method `ricecooker.classes.licenses.get_license` to create `Licence` objects. @@ -115,7 +115,7 @@ the stuff in the public domain. ### Languages The Python package `le-utils` defines the internal language codes used throughout the Kolibri platform (e.g. `en`, `es-MX`, and `zul`). To find the internal language -code for a given language, you can locate it in the [lookup table](https://github.com/learningequality/le-utils/blob/master/le_utils/resources/languagelookup.json), +code for a given language, you can locate it in the [lookup table](https://github.com/learningequality/le-utils/blob/main/le_utils/resources/languagelookup.json), or use one of the language lookup helper functions defined in `le_utils.constants.languages`: - `getlang() --> lang_obj`: basic lookup used to ensure `` is a valid internal language code (otherwise returns `None`). @@ -185,7 +185,7 @@ For your copy-paste convenience, here is the sample code for creating a content ) The pipeline infers the file(s), format preset, and content `kind` from `uri` — -see [files.md](./files.md) and [concepts/introduction.md](./concepts/introduction.md#kolibri-channels). +see [files.md](./files.md) and [concepts/introduction.md](./concepts/introduction.md#supported-content-kinds). `uri` accepts a URL or a local filesystem path, and the resource is downloaded for you automatically. @@ -313,7 +313,7 @@ They remain useful when you need to attach files the pipeline can't infer from a single `uri` alone — e.g. a `VideoNode` with an attached `SubtitleFile` (see [files.md](./files.md#subtitle-files)) — though the same effect is achieved more simply with `ContentNode(uri=...)` plus `add_file(SubtitleFile(...))`, -as shown in the [tutorial](../tutorial/tutorial.html#step-6-add-content). +as shown in the [tutorial](tutorial/tutorial.html#step-6-add-content). SlideshowNode nodes diff --git a/docs/tutorial/explanations.md b/docs/tutorial/explanations.md index 2c749004..73fbe034 100644 --- a/docs/tutorial/explanations.md +++ b/docs/tutorial/explanations.md @@ -139,7 +139,7 @@ On (8), we set the `language` attribute to the internal language code `en`, to i the content node is in English. The Python package `le-utils` defines the internal language codes used throughout the Kolibri platform (e.g. `en`, `es-MX`, and `zul`). To find the internal language code for a given language, you can locate it in the -[lookup table](https://github.com/learningequality/le-utils/blob/master/le_utils/resources/languagelookup.json), +[lookup table](https://github.com/learningequality/le-utils/blob/main/le_utils/resources/languagelookup.json), or use one of the language lookup helper functions defined in `le_utils.constants.languages`. Line (9) shows the `uri` attribute — the path or URL to the actual pdf file. `uri` can be diff --git a/docs/tutorial/gettingstarted.rst b/docs/tutorial/gettingstarted.rst index 2f7fd512..851128fe 100644 --- a/docs/tutorial/gettingstarted.rst +++ b/docs/tutorial/gettingstarted.rst @@ -116,7 +116,7 @@ If the command succeeds, you should see something like this printed in your term Validating channel structure... Potatoes info channel (ChannelNode): 2 descendants Potatoes! (TopicNode): 1 descendant - Growing potatoes (DocumentNode): 1 file + Growing potatoes (ContentNode): uri: https://www.gov.mb.ca/inr/pdf/pubs/mafri-potatoe.pdf Tree is valid Downloading files... diff --git a/docs/video_compression.md b/docs/video_compression.md index 23ee2246..b6bcf078 100644 --- a/docs/video_compression.md +++ b/docs/video_compression.md @@ -48,9 +48,6 @@ has the following transformations applied to it: - Scale the video to max-height of 480 pixels - Compress the video with CRF of 32 (constant rate factor) -To overwrite these defaults, chef authors can pass the argument `ffmpeg_settings` (dict), -when creating `VideoFile` object, and specify these options: `crf`, `max_height`, and `max_width`. - Manual conversion diff --git a/examples/tutorial/sushichef.py b/examples/tutorial/sushichef.py index 692d1353..1d71a117 100644 --- a/examples/tutorial/sushichef.py +++ b/examples/tutorial/sushichef.py @@ -76,7 +76,7 @@ def construct_channel(self, *args, **kwargs): title="Example Video", source_id="example-video", license=fancy_license, - uri="https://ia600209.us.archive.org/27/items/RiceChef/Rice Chef.mp4", + uri="https://archive.org/download/CM_National_Rice_Cooker_1982/CM_National_Rice_Cooker_1982.mp4", ) # uri alone can't express subtitles - add them explicitly, language is required examplevideo.add_file( From 98a237ff66828bb19f4ff11d6a3a9f3df00d0016 Mon Sep 17 00:00:00 2001 From: rtibblesbot Date: Sun, 5 Jul 2026 11:02:39 -0700 Subject: [PATCH 3/5] Tighten docs prose after PR #683 review pass Simplify wording across chefops, files, installation, nodes, and tutorial docs: separate staging/deploy into its own subsection, drop redundant explanations of uri inference, and point to AGENTS.md instead of duplicating the contributor quick start. Co-Authored-By: Claude Sonnet 5 --- docs/chefops.md | 11 ++++++----- docs/files.md | 5 ++--- docs/installation.md | 4 +--- docs/nodes.md | 17 ++++++++--------- docs/tutorial/explanations.md | 9 +++++---- 5 files changed, 22 insertions(+), 24 deletions(-) diff --git a/docs/chefops.md b/docs/chefops.md index 3752d738..79f4ca77 100644 --- a/docs/chefops.md +++ b/docs/chefops.md @@ -44,16 +44,17 @@ This listing shows the `ricecooker` command line interface (CLI) arguments: --publish Publish newly uploaded version of the channel. --sample SIZE Upload a sample of SIZE nodes from the channel. -**Staging is the default.** Ricecooker uploads a draft/staging tree by default, -so a chef run never overwrites the channel's live main tree unless you pass -`--deploy`. Use `--deploy` only when you intentionally want to overwrite the -main tree. - As you can tell, there are lot of arguments to choose from, and this is not even the complete list: you'll have to run `./sushichef.py -h` to see the latest version. Below is a short guide to some of the most important and useful ones arguments. +### Staging vs. deploy +Ricecooker uploads a draft/staging tree by default, so a chef run never +overwrites the channel's live main tree unless you pass `--deploy`. Use +`--deploy` only when you intentionally want to overwrite the main tree. + + ### Compression and thumbnail globals You can specify video compression settings (see this page) and thumbnails for specific nodes and files in the channel, or use `--compress` and `--thumbnails` diff --git a/docs/files.md b/docs/files.md index 977664b3..4ec70c2d 100644 --- a/docs/files.md +++ b/docs/files.md @@ -112,9 +112,8 @@ video_node.add_file(YouTubeSubtitleFile(youtube_id=youtube_id, language="en")) Legacy / advanced: constructing file objects directly ------------------------------------------------------ -These classes are what the pipeline builds internally from `uri` — construct -them directly only when you need `add_file()` for a case `uri` alone can't -express, or need pipeline-internal details. +Construct these classes directly only for the exceptions noted above +(exercises, subtitles) or when you need pipeline-internal detail. ### Audio files Use the `AudioFile(DownloadFile)` class to store `mp3` files. diff --git a/docs/installation.md b/docs/installation.md index 399206b6..4f6b0d31 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -95,9 +95,7 @@ dependency in `pyproject.toml`. Contributing to ricecooker itself ---------------------------------- If you're contributing to `ricecooker` itself (not just writing a chef script), -clone the repo and run `uv sync --group dev` to install all dev dependencies, -`uv run --group test pytest` to run the test suite, and `uvx prek run --all-files` -to lint before committing — see `AGENTS.md` for the full contributor quick start. +see `AGENTS.md` for the contributor quick start. **Reporting issues**: If you run into problems or encounter an error in any of the above steps, please let us know by [opening an issue on github](https://github.com/learningequality/ricecooker/issues). diff --git a/docs/nodes.md b/docs/nodes.md index 68b1b405..123a02ea 100644 --- a/docs/nodes.md +++ b/docs/nodes.md @@ -184,10 +184,9 @@ For your copy-paste convenience, here is the sample code for creating a content uri='https://www.gov.mb.ca/inr/pdf/pubs/mafri-potatoe.pdf', ) -The pipeline infers the file(s), format preset, and content `kind` from `uri` — -see [files.md](./files.md) and [concepts/introduction.md](./concepts/introduction.md#supported-content-kinds). -`uri` accepts a URL or a local filesystem path, and the resource is downloaded -for you automatically. +`uri` accepts a URL or a local filesystem path — the pipeline downloads it and +infers everything else automatically, as described in +[concepts/introduction.md](./concepts/introduction.md#supported-content-kinds). Specify `derive_thumbnail=True` and leave thumbnail blank (`thumbnail=None`) to let Ricecooker automatically generate a thumbnail for the node based on its content. @@ -309,11 +308,11 @@ subclasses used to require you to specify by hand. These classes remain supported and are not scheduled for removal, but are no longer the documented happy path. -They remain useful when you need to attach files the pipeline can't infer -from a single `uri` alone — e.g. a `VideoNode` with an attached `SubtitleFile` -(see [files.md](./files.md#subtitle-files)) — though the same effect is -achieved more simply with `ContentNode(uri=...)` plus `add_file(SubtitleFile(...))`, -as shown in the [tutorial](tutorial/tutorial.html#step-6-add-content). +There is no remaining functional advantage to using these classes over +`ContentNode(uri=...)` — even the subtitle case (see +[files.md](./files.md#subtitle-files)) is expressed more simply as +`ContentNode(uri=...)` plus `add_file(SubtitleFile(...))`, as shown in the +[tutorial](tutorial/tutorial.html#step-6-add-content). SlideshowNode nodes diff --git a/docs/tutorial/explanations.md b/docs/tutorial/explanations.md index 73fbe034..7d6b67b4 100644 --- a/docs/tutorial/explanations.md +++ b/docs/tutorial/explanations.md @@ -119,10 +119,11 @@ you to a structure that best first the needs of learners and teachers. ### Content nodes The canonical way to add a content item is `ContentNode(source_id, title, license, uri=...)`. -Passing a local path or URL as `uri` hands it to the file pipeline (transfer → convert → -extract_metadata), which infers the content's kind and format preset and builds the -underlying file object automatically — you don't need to construct a file object yourself -for the vast majority of content. +Passing a local path or URL as `uri` downloads the resource and infers its kind, format +preset, and underlying file object automatically — see +[concepts/introduction.md](../concepts/introduction.md#supported-content-kinds) for how +this inference works. You don't need to construct a file object yourself for the vast +majority of content. Line (6) shows how to create a `ContentNode` to store the metadata for a pdf file. The `title` and `description` attributes are set, and `source_id` is set to a unique From 0afc603ec596648ee3cccebff32b07c67b3cc449 Mon Sep 17 00:00:00 2001 From: rtibblesbot Date: Sun, 5 Jul 2026 14:36:52 -0700 Subject: [PATCH 4/5] Address second round of PR #683 review feedback - ContentNode now passes a `context` dict through to the file pipeline and inherits the node's `language` onto files it builds from `uri`, fixing an inconsistency the reviewer flagged where a file built from `uri` had no language unless the pipeline inferred one itself. - docs/files.md: point YouTube-subtitle docs at `context={"subtitle_languages": [...]}` instead of a manual `YouTubeSubtitleFile.add_file` call, since that's the mechanism that actually triggers automatic subtitle download from the `ContentNode(uri=)` path. - docs/tutorial/gettingstarted.rst: drop the "Video overview" section and its dead video embed. Co-Authored-By: Claude Sonnet 5 --- docs/files.md | 9 +++++---- docs/tutorial/explanations.md | 2 ++ docs/tutorial/gettingstarted.rst | 11 ----------- ricecooker/classes/nodes.py | 12 ++++++++++-- tests/test_tree.py | 31 +++++++++++++++++++++++++++++++ 5 files changed, 48 insertions(+), 17 deletions(-) diff --git a/docs/files.md b/docs/files.md index 4ec70c2d..92c8dac3 100644 --- a/docs/files.md +++ b/docs/files.md @@ -97,16 +97,17 @@ a given youtube language code is supported by `YouTubeSubtitleFile` and skip the ones that are not currently supported. Please let the LE content team know when you run into language codes that are not supported so we can add them. -`YouTubeSubtitleFile` pairs with the `ContentNode(uri=...)` approach for YouTube -videos: passing a YouTube URL as `uri` downloads the video, and `YouTubeSubtitleFile` -downloads the caption track directly from YouTube — no local `.vtt` file needed: +For the `ContentNode(uri=...)` approach, pass `subtitle_languages` in the `context` +argument instead — it triggers the same YouTube caption download alongside the +video, with no explicit `YouTubeSubtitleFile` or local `.vtt` file needed: ``` video_node = ContentNode( source_id, title, license, uri=f"https://www.youtube.com/watch?v={youtube_id}", + context={"subtitle_languages": ["en", "es"]}, ) -video_node.add_file(YouTubeSubtitleFile(youtube_id=youtube_id, language="en")) ``` +See [`YoutubeDownloadHandler`](https://github.com/learningequality/ricecooker/blob/main/ricecooker/utils/pipeline/transfer.py#L172) in the pipeline for how `subtitle_languages` is consumed. diff --git a/docs/tutorial/explanations.md b/docs/tutorial/explanations.md index 7d6b67b4..b481fc04 100644 --- a/docs/tutorial/explanations.md +++ b/docs/tutorial/explanations.md @@ -142,6 +142,8 @@ language codes used throughout the Kolibri platform (e.g. `en`, `es-MX`, and `zu To find the internal language code for a given language, you can locate it in the [lookup table](https://github.com/learningequality/le-utils/blob/main/le_utils/resources/languagelookup.json), or use one of the language lookup helper functions defined in `le_utils.constants.languages`. +This `language` is also inherited by the file the pipeline builds from `uri` — you +don't need to set it again on a file object. Line (9) shows the `uri` attribute — the path or URL to the actual pdf file. `uri` can be either a local filesystem path or a web URL (as in the above example); URLs are downloaded diff --git a/docs/tutorial/gettingstarted.rst b/docs/tutorial/gettingstarted.rst index 851128fe..a974b9df 100644 --- a/docs/tutorial/gettingstarted.rst +++ b/docs/tutorial/gettingstarted.rst @@ -35,17 +35,6 @@ and pass in the command line argument ``--token=path/to/studiotoken.txt``. -Video overview --------------- -Watch this video tutorial to learn how to create a new content integration script and set the required -channel metadata fields like ``CHANNEL_SOURCE_DOMAIN`` and ``CHANNEL_SOURCE_ID``. - -.. raw:: html - - -
 
- - Creating a sushichef script --------------------------- In a new folder on your computer, create a file called ``sushichef.py`` with the diff --git a/ricecooker/classes/nodes.py b/ricecooker/classes/nodes.py index 43731d66..87519b30 100644 --- a/ricecooker/classes/nodes.py +++ b/ricecooker/classes/nodes.py @@ -826,15 +826,20 @@ class ContentNode(TreeNode): Attributes: uri (str): URI for the main file for this content node (optional) pipeline (FilePipeline): FilePipeline instance for handling uri processing (optional) + context (dict): extra context passed to the pipeline when processing uri, e.g. + {"subtitle_languages": ["en", "es"]} to also fetch YouTube subtitles (optional) See TreeNode and Node for inherited attributes. """ required_presets = tuple() - def __init__(self, source_id, title, license, uri=None, pipeline=None, **kwargs): + def __init__( + self, source_id, title, license, uri=None, pipeline=None, context=None, **kwargs + ): self.uri = uri self._pipeline = pipeline + self.context = context or {} # Flag here to say that files haven't been processed. # Until files have been processed we can't be sure that the files are actually valid # for example, once we download a file we may discover it doesn't exist, that it's @@ -916,7 +921,7 @@ def _validate(self): def _process_uri(self): try: file_metadata_list = self.pipeline.execute( - self.uri, skip_cache=config.UPDATE + self.uri, context=self.context, skip_cache=config.UPDATE ) except (InvalidFileException, ExpectedFileException) as e: config.LOGGER.error(f"Error processing path: {self.uri} with error: {e}") @@ -928,6 +933,9 @@ def _process_uri(self): content_metadata.update(metadata_dict.pop("content_node_metadata")) # Remove path from metadata_dict as it is not needed for the File object metadata_dict.pop("path", None) + # Inherit the node's language unless the pipeline inferred one of its own + # (e.g. a subtitle language extracted from the file itself). + metadata_dict.setdefault("language", self.language) file_obj = File(**metadata_dict) self.add_file(file_obj) for key, value in content_metadata.items(): diff --git a/tests/test_tree.py b/tests/test_tree.py index 1aea7f11..2c6381c2 100644 --- a/tests/test_tree.py +++ b/tests/test_tree.py @@ -747,6 +747,37 @@ def test_automatic_resource_node_document(document_file): assert node.learning_activities == [learning_activities.READ] +def test_automatic_resource_node_document_inherits_node_language(document_file): + node = ContentNode( + "test", + "test", + licenses.CC_BY, + uri=document_file.path, + pipeline=FilePipeline(), + copyright_holder="Demo Holdings", + language="en", + ) + node.process_files() + assert all(f.language == "en" for f in node.files) + + +def test_content_node_passes_context_to_pipeline(): + mock_pipeline = MagicMock() + mock_pipeline.execute.return_value = [] + node = ContentNode( + "test", + "test", + licenses.CC_BY, + uri="https://www.youtube.com/watch?v=abcdefghijk", + pipeline=mock_pipeline, + context={"subtitle_languages": ["en", "es"]}, + ) + node._process_uri() + mock_pipeline.execute.assert_called_once_with( + node.uri, context={"subtitle_languages": ["en", "es"]}, skip_cache=False + ) + + def test_automatic_resource_node_epub(epub_file): node = ContentNode( "test", From 5ffe922ca0b2c902fefc02acd5390dbbb04b2896 Mon Sep 17 00:00:00 2001 From: rtibblesbot Date: Sun, 5 Jul 2026 14:47:08 -0700 Subject: [PATCH 5/5] Tighten docs prose after PR #683 review pass Remove duplicated/redundant explanations of staging-vs-deploy and uri-based file inference now that the canonical explanation lives in one place, and point readers there instead. Co-Authored-By: Claude Sonnet 5 --- docs/chefops.md | 10 +++------- docs/nodes.md | 5 ++--- docs/tutorial/explanations.md | 5 ++--- 3 files changed, 7 insertions(+), 13 deletions(-) diff --git a/docs/chefops.md b/docs/chefops.md index 79f4ca77..3b26890a 100644 --- a/docs/chefops.md +++ b/docs/chefops.md @@ -40,7 +40,9 @@ This listing shows the `ricecooker` command line interface (CLI) arguments: --prompt Prompt user to open the channel after the chef run. --deploy Immediately deploy changes to channel's main tree. This operation will overwrite the previous channel - content. Use only during development. + content. Use only during development. Staging is + the default, so omit this flag to get a safe, + reviewable draft upload instead. --publish Publish newly uploaded version of the channel. --sample SIZE Upload a sample of SIZE nodes from the channel. @@ -49,12 +51,6 @@ the complete list: you'll have to run `./sushichef.py -h` to see the latest vers Below is a short guide to some of the most important and useful ones arguments. -### Staging vs. deploy -Ricecooker uploads a draft/staging tree by default, so a chef run never -overwrites the channel's live main tree unless you pass `--deploy`. Use -`--deploy` only when you intentionally want to overwrite the main tree. - - ### Compression and thumbnail globals You can specify video compression settings (see this page) and thumbnails for specific nodes and files in the channel, or use `--compress` and `--thumbnails` diff --git a/docs/nodes.md b/docs/nodes.md index 123a02ea..31a32494 100644 --- a/docs/nodes.md +++ b/docs/nodes.md @@ -184,9 +184,8 @@ For your copy-paste convenience, here is the sample code for creating a content uri='https://www.gov.mb.ca/inr/pdf/pubs/mafri-potatoe.pdf', ) -`uri` accepts a URL or a local filesystem path — the pipeline downloads it and -infers everything else automatically, as described in -[concepts/introduction.md](./concepts/introduction.md#supported-content-kinds). +See the [Overview](#overview) above for how the pipeline infers kind, preset, +and files from `uri`. Specify `derive_thumbnail=True` and leave thumbnail blank (`thumbnail=None`) to let Ricecooker automatically generate a thumbnail for the node based on its content. diff --git a/docs/tutorial/explanations.md b/docs/tutorial/explanations.md index b481fc04..057db221 100644 --- a/docs/tutorial/explanations.md +++ b/docs/tutorial/explanations.md @@ -152,9 +152,8 @@ is to cache downloaded files forever. Use the `--update` argument to bypass the re-download all files. The `--update` must be used whenever files are modified but the path stays the same. -The only cases `uri` alone can't express are exercises (`ExerciseNode` + questions) and -video subtitles (`SubtitleFile`, which requires an explicit `language`) — see -[Legacy / advanced node classes](../nodes.md#legacy-advanced-node-classes) and +The two exceptions `uri` alone can't express are exercises and video subtitles — +see [Legacy / advanced node classes](../nodes.md#legacy-advanced-node-classes) and [Subtitle files](../files.md#subtitle-files).