Try Airflow
In [ ]:
%%bash
# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
mkdir ~/airflow
%env AIRFLOW_HOME=~/airflow
In [ ]:
!sudo -HE pip install marshmallow==2.18.0
Collecting marshmallow==2.18.0
Downloading https://files.pythonhosted.org/packages/9a/3c/4cc463c53136dc503f9ee234c4a6592e1c7411cb362f844e80df70361f29/marshmallow-2.18.0-py2.py3-none-any.whl (50kB)
|████████████████████████████████| 51kB 15.9MB/s eta 0:00:01
Installing collected packages: marshmallow
Found existing installation: marshmallow 3.2.2
Uninstalling marshmallow-3.2.2:
Successfully uninstalled marshmallow-3.2.2
Successfully installed marshmallow-2.18.0
In [ ]:
%%bash
# install from pypi using pip
sudo -HE pip install apache-airflow
Requirement already satisfied: apache-airflow in /opt/tljh/user/lib/python3.6/site-packages (1.10.6) Requirement already satisfied: cached-property~=1.5 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.5.1) Requirement already satisfied: future<0.17,>=0.16.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.16.0) Requirement already satisfied: text-unidecode==1.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.2) Requirement already satisfied: unicodecsv>=0.14.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.14.1) Requirement already satisfied: iso8601>=0.1.12 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.1.12) Requirement already satisfied: croniter<0.4,>=0.3.17 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.3.30) Requirement already satisfied: flask-login<0.5,>=0.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.4.1) Requirement already satisfied: python-dateutil<3,>=2.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.8.0) Requirement already satisfied: requests<3,>=2.20.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.22.0) Requirement already satisfied: colorlog==4.0.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (4.0.2) Requirement already satisfied: dill<0.4,>=0.2.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.3.1.1) Requirement already satisfied: python-daemon<2.2,>=2.1.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.1.2) Requirement already satisfied: marshmallow-sqlalchemy<0.19.0,>=0.16.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.18.0) Requirement already satisfied: psutil<6.0.0,>=4.2.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (5.6.3) Requirement already satisfied: termcolor==1.1.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.1.0) Requirement already satisfied: flask-admin==1.5.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.5.3) Requirement already satisfied: flask-swagger==0.2.13 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.2.13) Requirement already satisfied: pygments<3.0,>=2.0.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.4.2) Requirement already satisfied: sqlalchemy~=1.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.3.8) Requirement already satisfied: funcsigs==1.0.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.0.0) Requirement already satisfied: flask<2.0,>=1.1.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.1.1) Requirement already satisfied: tenacity==4.12.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (4.12.0) Requirement already satisfied: setproctitle<2,>=1.1.8 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.1.10) Requirement already satisfied: thrift>=0.9.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.13.0) Requirement already satisfied: pendulum==1.4.4 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.4.4) Requirement already satisfied: jinja2<2.11.0,>=2.10.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.10.1) Requirement already satisfied: gunicorn<20.0,>=19.5.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (19.9.0) Requirement already satisfied: argcomplete~=1.10 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.10.3) Requirement already satisfied: flask-appbuilder<2.0.0,>=1.12.5 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.13.1) Requirement already satisfied: lazy-object-proxy~=1.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.4.3) Requirement already satisfied: json-merge-patch==0.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.2) Requirement already satisfied: flask-wtf<0.15,>=0.14.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.14.2) Requirement already satisfied: tabulate<0.9,>=0.7.5 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.8.6) Requirement already satisfied: flask-caching<1.4.0,>=1.3.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.3.3) Requirement already satisfied: configparser<3.6.0,>=3.5.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (3.5.3) Requirement already satisfied: zope.deprecation<5.0,>=4.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (4.4.0) Requirement already satisfied: markdown<3.0,>=2.5.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.6.11) Requirement already satisfied: pandas<1.0.0,>=0.17.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.25.1) Requirement already satisfied: alembic<2.0,>=1.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.1.0) Requirement already satisfied: tzlocal<2.0.0,>=1.4 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.5.1) Requirement already satisfied: graphviz>=0.12 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.13.2) Requirement already satisfied: six>=1.5 in /opt/tljh/user/lib/python3.6/site-packages (from python-dateutil<3,>=2.3->apache-airflow) (1.12.0) Requirement already satisfied: certifi>=2017.4.17 in /opt/tljh/user/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow) (2019.9.11) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/tljh/user/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow) (1.25.3) Requirement already satisfied: idna<2.9,>=2.5 in /opt/tljh/user/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow) (2.8) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/tljh/user/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow) (3.0.4) Requirement already satisfied: setuptools in /opt/tljh/user/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow) (41.4.0) Requirement already satisfied: docutils in /opt/tljh/user/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow) (0.15.2) Requirement already satisfied: lockfile>=0.10 in /opt/tljh/user/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow) (0.12.2) Requirement already satisfied: marshmallow>=2.0.0 in /opt/tljh/user/lib/python3.6/site-packages (from marshmallow-sqlalchemy<0.19.0,>=0.16.1->apache-airflow) (2.18.0) Requirement already satisfied: wtforms in /opt/tljh/user/lib/python3.6/site-packages (from flask-admin==1.5.3->apache-airflow) (2.2.1) Requirement already satisfied: PyYAML>=3.0 in /opt/tljh/user/lib/python3.6/site-packages (from flask-swagger==0.2.13->apache-airflow) (5.1.2) Requirement already satisfied: ordereddict in /opt/tljh/user/lib/python3.6/site-packages (from funcsigs==1.0.0->apache-airflow) (1.1) Requirement already satisfied: itsdangerous>=0.24 in /opt/tljh/user/lib/python3.6/site-packages (from flask<2.0,>=1.1.0->apache-airflow) (1.1.0) Requirement already satisfied: Werkzeug>=0.15 in /opt/tljh/user/lib/python3.6/site-packages (from flask<2.0,>=1.1.0->apache-airflow) (0.16.0) Requirement already satisfied: click>=5.1 in /opt/tljh/user/lib/python3.6/site-packages (from flask<2.0,>=1.1.0->apache-airflow) (7.0) Requirement already satisfied: pytzdata>=2018.3.0.0 in /opt/tljh/user/lib/python3.6/site-packages (from pendulum==1.4.4->apache-airflow) (2019.3) Requirement already satisfied: MarkupSafe>=0.23 in /opt/tljh/user/lib/python3.6/site-packages (from jinja2<2.11.0,>=2.10.1->apache-airflow) (1.1.1) Requirement already satisfied: apispec[yaml]>=1.1.1<2 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (3.1.0) Requirement already satisfied: Flask-OpenID<2,>=1.2.5 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (1.2.5) Requirement already satisfied: marshmallow-enum<2,>=1.4.1 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (1.5.1) Requirement already satisfied: colorama<1,>=0.3.9 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.4.1) Requirement already satisfied: Flask-JWT-Extended<4,>=3.18 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (3.24.1) Requirement already satisfied: prison==0.1.0 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.1.0) Requirement already satisfied: Flask-Babel<1,>=0.11.1 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.12.2) Requirement already satisfied: jsonschema>=3.0.1<4 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (3.0.2) Requirement already satisfied: PyJWT>=1.7.1 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (1.7.1) Requirement already satisfied: Flask-SQLAlchemy<3,>=2.3 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (2.4.1) Requirement already satisfied: numpy>=1.13.3 in /opt/tljh/user/lib/python3.6/site-packages (from pandas<1.0.0,>=0.17.1->apache-airflow) (1.17.2) Requirement already satisfied: pytz>=2017.2 in /opt/tljh/user/lib/python3.6/site-packages (from pandas<1.0.0,>=0.17.1->apache-airflow) (2019.2) Requirement already satisfied: Mako in /opt/tljh/user/lib/python3.6/site-packages (from alembic<2.0,>=1.0->apache-airflow) (1.1.0) Requirement already satisfied: python-editor>=0.3 in /opt/tljh/user/lib/python3.6/site-packages (from alembic<2.0,>=1.0->apache-airflow) (1.0.4) Requirement already satisfied: python3-openid>=2.0 in /opt/tljh/user/lib/python3.6/site-packages (from Flask-OpenID<2,>=1.2.5->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (3.1.0) Requirement already satisfied: Babel>=2.3 in /opt/tljh/user/lib/python3.6/site-packages (from Flask-Babel<1,>=0.11.1->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (2.7.0) Requirement already satisfied: attrs>=17.4.0 in /opt/tljh/user/lib/python3.6/site-packages (from jsonschema>=3.0.1<4->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (19.1.0) Requirement already satisfied: pyrsistent>=0.14.0 in /opt/tljh/user/lib/python3.6/site-packages (from jsonschema>=3.0.1<4->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.15.4) Requirement already satisfied: defusedxml in /opt/tljh/user/lib/python3.6/site-packages (from python3-openid>=2.0->Flask-OpenID<2,>=1.2.5->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.6.0)
In [ ]:
%%bash
# initialize the database
airflow initdb
DB: sqlite:////home/jupyter-sheng_wei/airflow/airflow.db
[2019-12-02 14:25:56,971] {db.py:368} INFO - Creating tables
Done.
INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current schema INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted /opt/tljh/user/lib/python3.6/site-packages/alembic/ddl/sqlite.py:39: UserWarning: Skipping unsupported ALTER for creation of implicit constraint "Skipping unsupported ALTER for " INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_instance INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices INFO [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding extra to Log INFO [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, add dagrun INFO [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, task_duration INFO [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, dagrun_config INFO [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, add password column to user INFO [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, dagrun start end INFO [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss INFO [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection INFO [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table INFO [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, add TI state index INFO [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, add task fails journal table INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, add dag_stats table INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, xcom dag task indices INFO [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table INFO [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> bdaa763e6c56, Make xcom value column a large binary INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> 947454bf1dff, add ti job_id index INFO [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text types) INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 33ae817a1ff4, kubernetes_resource_checkpointing INFO [alembic.runtime.migration] Running upgrade 33ae817a1ff4 -> 27c6a30d7c24, kubernetes_resource_checkpointing INFO [alembic.runtime.migration] Running upgrade 27c6a30d7c24 -> 86770d1215c0, add kubernetes scheduler uniqueness INFO [alembic.runtime.migration] Running upgrade 86770d1215c0, 0e2a74e0fc9f -> 05f30312d566, merge heads INFO [alembic.runtime.migration] Running upgrade 05f30312d566 -> f23433877c24, fix mysql not null constraint INFO [alembic.runtime.migration] Running upgrade f23433877c24 -> 856955da8476, fix sqlite foreign key INFO [alembic.runtime.migration] Running upgrade 856955da8476 -> 9635ae0956e7, index-faskfail INFO [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> dd25f486b8ea, add idx_log_dag INFO [alembic.runtime.migration] Running upgrade dd25f486b8ea -> bf00311e1990, add index to taskinstance INFO [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> 0a2a5b66e19d, add task_reschedule table INFO [alembic.runtime.migration] Running upgrade 0a2a5b66e19d, bf00311e1990 -> 03bc53e68815, merge_heads_2 INFO [alembic.runtime.migration] Running upgrade 03bc53e68815 -> 41f5f12752f8, add superuser field INFO [alembic.runtime.migration] Running upgrade 41f5f12752f8 -> c8ffec048a3b, add fields to dag INFO [alembic.runtime.migration] Running upgrade c8ffec048a3b -> dd4ecb8fbee3, Add schedule interval to dag INFO [alembic.runtime.migration] Running upgrade dd4ecb8fbee3 -> 939bb1e647c8, task reschedule fk on cascade delete INFO [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 6e96a59344a4, Make TaskInstance.pool not nullable INFO [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> 74effc47d867, change datetime to datetime2(6) on MSSQL tables INFO [alembic.runtime.migration] Running upgrade c8ffec048a3b -> a56c9515abdc, Remove dag_stat table INFO [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, increase queue name size limit
In [ ]:
%%bash
# start the web server, default port is 8080
airflow webserver -p 8080
bash: fg: %%bash: no such job ____________ _____________ ____ |__( )_________ __/__ /________ __ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / / ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ [2019-12-02 14:30:48,749] {__init__.py:51} INFO - Using executor SequentialExecutor [2019-12-02 14:30:48,750] {dagbag.py:92} INFO - Filling up the DagBag from /home/jupyter-sheng_wei/airflow/dags Running the Gunicorn Server with: Workers: 4 sync Host: 0.0.0.0:8080 Timeout: 120 Logfiles: - - ================================================================= [2019-12-02 14:30:49 +0900] [14677] [INFO] Starting gunicorn 19.9.0 [2019-12-02 14:30:49 +0900] [14677] [INFO] Listening at: http://0.0.0.0:8080 (14677) [2019-12-02 14:30:49 +0900] [14677] [INFO] Using worker: sync [2019-12-02 14:30:49 +0900] [14682] [INFO] Booting worker with pid: 14682 [2019-12-02 14:30:50 +0900] [14683] [INFO] Booting worker with pid: 14683 [2019-12-02 14:30:50 +0900] [14684] [INFO] Booting worker with pid: 14684 [2019-12-02 14:30:50 +0900] [14685] [INFO] Booting worker with pid: 14685 [2019-12-02 14:30:50,996] {__init__.py:51} INFO - Using executor SequentialExecutor [2019-12-02 14:30:51,008] {dagbag.py:92} INFO - Filling up the DagBag from /home/jupyter-sheng_wei/airflow/dags [2019-12-02 14:30:51,289] {__init__.py:51} INFO - Using executor SequentialExecutor [2019-12-02 14:30:51,304] {dagbag.py:92} INFO - Filling up the DagBag from /home/jupyter-sheng_wei/airflow/dags [2019-12-02 14:30:51,442] {__init__.py:51} INFO - Using executor SequentialExecutor [2019-12-02 14:30:51,456] {dagbag.py:92} INFO - Filling up the DagBag from /home/jupyter-sheng_wei/airflow/dags [2019-12-02 14:30:51,469] {__init__.py:51} INFO - Using executor SequentialExecutor [2019-12-02 14:30:51,480] {dagbag.py:92} INFO - Filling up the DagBag from /home/jupyter-sheng_wei/airflow/dags 123.226.242.94 - - [02/Dec/2019:14:31:07 +0900] "GET / HTTP/1.1" 302 221 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36" 123.226.242.94 - - [02/Dec/2019:14:31:08 +0900] "GET /admin/ HTTP/1.1" 200 115313 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36" 123.226.242.94 - - [02/Dec/2019:14:31:08 +0900] "GET /admin/airflow/blocked HTTP/1.1" 200 2 "http://jhub.name:8080/admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36" 123.226.242.94 - - [02/Dec/2019:14:31:08 +0900] "GET /admin/airflow/dag_stats HTTP/1.1" 200 9486 "http://jhub.name:8080/admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36" 123.226.242.94 - - [02/Dec/2019:14:31:08 +0900] "GET /admin/airflow/task_stats HTTP/1.1" 200 30510 "http://jhub.name:8080/admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36" [2019-12-02 14:31:23 +0900] [14677] [INFO] Handling signal: ttin [2019-12-02 14:31:23 +0900] [14687] [INFO] Booting worker with pid: 14687 [2019-12-02 14:31:23,855] {__init__.py:51} INFO - Using executor SequentialExecutor [2019-12-02 14:31:23,856] {dagbag.py:92} INFO - Filling up the DagBag from /home/jupyter-sheng_wei/airflow/dags [2019-12-02 14:31:24 +0900] [14677] [INFO] Handling signal: ttou [2019-12-02 14:31:24 +0900] [14682] [INFO] Worker exiting (pid: 14682) [2019-12-02 14:31:30 +0900] [14677] [INFO] Handling signal: int [2019-12-02 14:31:30 +0900] [14683] [INFO] Worker exiting (pid: 14683) [2019-12-02 14:31:30 +0900] [14684] [INFO] Worker exiting (pid: 14684) [2019-12-02 14:31:30 +0900] [14685] [INFO] Worker exiting (pid: 14685) [2019-12-02 14:31:30 +0900] [14687] [INFO] Worker exiting (pid: 14687) [2019-12-02 14:31:31 +0900] [14677] [INFO] Shutting down: Master
In [ ]:
# start the scheduler
airflow scheduler
____________ _____________ ____ |__( )_________ __/__ /________ __ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / / ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ [2019-12-02 14:33:56,327] {__init__.py:51} INFO - Using executor SequentialExecutor [2019-12-02 14:33:56,343] {scheduler_job.py:1320} INFO - Starting the scheduler [2019-12-02 14:33:56,343] {scheduler_job.py:1328} INFO - Running execute loop for -1 seconds [2019-12-02 14:33:56,345] {scheduler_job.py:1329} INFO - Processing each file at most -1 times [2019-12-02 14:33:56,346] {scheduler_job.py:1332} INFO - Searching for files in /home/jupyter-sheng_wei/airflow/dags [2019-12-02 14:33:56,351] {scheduler_job.py:1334} INFO - There are 20 files in /home/jupyter-sheng_wei/airflow/dags [2019-12-02 14:33:56,354] {scheduler_job.py:1382} INFO - Resetting orphaned tasks for active dag runs [2019-12-02 14:33:56,366] {dag_processing.py:556} INFO - Launched DagFileProcessorManager with pid: 14781 [2019-12-02 14:33:56,374] {settings.py:54} INFO - Configured default timezone <Timezone [UTC]> [2019-12-02 14:33:56,384] {dag_processing.py:760} ERROR - Cannot use more than 1 thread when using sqlite. Setting parallelism to 1 [2019-12-02 14:34:11,631] {helpers.py:308} INFO - Sending Signals.SIGTERM to GPID 14781 [2019-12-02 14:34:11,636] {helpers.py:286} INFO - Process psutil.Process(pid=14781, status='terminated') (14781) terminated with exit code 0 [2019-12-02 14:34:11,637] {scheduler_job.py:1361} INFO - Exited execute loop