Andrew's Blog

PURSUE ULTIMATE FACTS

2019-10-19
Learn BigQuery

https://ipython.readthedocs.io/en/stable/interactive/magics.html

In [ ]:
%%bash
pip install --upgrade 'google-cloud-bigquery[bqstorage,pandas]'

Collecting google-cloud-bigquery[bqstorage,pandas]
  Downloading google_cloud_bigquery-2.30.1-py2.py3-none-any.whl (203 kB)
Collecting proto-plus>=1.10.0
  Downloading proto_plus-1.19.8-py3-none-any.whl (45 kB)
Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.9/site-packages (from google-cloud-bigquery[bqstorage,pandas]) (21.2)
Requirement already satisfied: protobuf>=3.12.0 in /opt/conda/lib/python3.9/site-packages (from google-cloud-bigquery[bqstorage,pandas]) (3.18.1)
Collecting google-api-core[grpc]<3.0.0dev,>=1.29.0
  Downloading google_api_core-2.2.2-py2.py3-none-any.whl (95 kB)
Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /opt/conda/lib/python3.9/site-packages (from google-cloud-bigquery[bqstorage,pandas]) (2.8.2)
Collecting grpcio<2.0dev,>=1.38.1
  Downloading grpcio-1.42.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.0 MB)
Collecting google-resumable-media<3.0dev,>=0.6.0
  Downloading google_resumable_media-2.1.0-py2.py3-none-any.whl (75 kB)
Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.9/site-packages (from google-cloud-bigquery[bqstorage,pandas]) (2.26.0)
Collecting google-cloud-core<3.0.0dev,>=1.4.1
  Downloading google_cloud_core-2.2.1-py2.py3-none-any.whl (29 kB)
Requirement already satisfied: pyarrow<7.0dev,>=3.0.0 in /opt/conda/lib/python3.9/site-packages (from google-cloud-bigquery[bqstorage,pandas]) (6.0.0)
Collecting google-cloud-bigquery-storage<3.0.0dev,>=2.0.0
  Downloading google_cloud_bigquery_storage-2.10.1-py2.py3-none-any.whl (171 kB)
Requirement already satisfied: pandas>=0.24.2 in /opt/conda/lib/python3.9/site-packages (from google-cloud-bigquery[bqstorage,pandas]) (1.3.4)
Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.9/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.29.0->google-cloud-bigquery[bqstorage,pandas]) (59.1.1)
Collecting googleapis-common-protos<2.0dev,>=1.52.0
  Downloading googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting google-auth<3.0dev,>=1.25.0
  Downloading google_auth-2.3.3-py2.py3-none-any.whl (155 kB)
Collecting grpcio-status<2.0dev,>=1.33.2
  Downloading grpcio_status-1.42.0-py3-none-any.whl (10.0 kB)
Collecting libcst>=0.2.5
  Downloading libcst-0.3.23-py3-none-any.whl (517 kB)
Collecting google-crc32c<2.0dev,>=1.0
  Downloading google_crc32c-1.3.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (36 kB)
Requirement already satisfied: six>=1.5.2 in /opt/conda/lib/python3.9/site-packages (from grpcio<2.0dev,>=1.38.1->google-cloud-bigquery[bqstorage,pandas]) (1.16.0)
Requirement already satisfied: pyparsing<3,>=2.0.2 in /opt/conda/lib/python3.9/site-packages (from packaging>=14.3->google-cloud-bigquery[bqstorage,pandas]) (2.4.7)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.9/site-packages (from pandas>=0.24.2->google-cloud-bigquery[bqstorage,pandas]) (2021.3)
Requirement already satisfied: numpy>=1.17.3 in /opt/conda/lib/python3.9/site-packages (from pandas>=0.24.2->google-cloud-bigquery[bqstorage,pandas]) (1.20.3)
Collecting protobuf>=3.12.0
  Downloading protobuf-3.19.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.9/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-bigquery[bqstorage,pandas]) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.9/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-bigquery[bqstorage,pandas]) (1.26.7)
Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.9/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-bigquery[bqstorage,pandas]) (2.0.0)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.9/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-bigquery[bqstorage,pandas]) (3.1)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.8-py3-none-any.whl (39 kB)
Collecting cachetools<5.0,>=2.0.0
  Downloading cachetools-4.2.4-py3-none-any.whl (10 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting typing-inspect>=0.4.0
  Downloading typing_inspect-0.7.1-py3-none-any.whl (8.4 kB)
Requirement already satisfied: typing-extensions>=3.7.4.2 in /opt/conda/lib/python3.9/site-packages (from libcst>=0.2.5->google-cloud-bigquery-storage<3.0.0dev,>=2.0.0->google-cloud-bigquery[bqstorage,pandas]) (3.10.0.2)
Requirement already satisfied: pyyaml>=5.2 in /opt/conda/lib/python3.9/site-packages (from libcst>=0.2.5->google-cloud-bigquery-storage<3.0.0dev,>=2.0.0->google-cloud-bigquery[bqstorage,pandas]) (6.0)
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting mypy-extensions>=0.3.0
  Downloading mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Installing collected packages: pyasn1, rsa, pyasn1-modules, protobuf, cachetools, mypy-extensions, grpcio, googleapis-common-protos, google-auth, typing-inspect, grpcio-status, google-crc32c, google-api-core, proto-plus, libcst, google-resumable-media, google-cloud-core, google-cloud-bigquery-storage, google-cloud-bigquery
  Attempting uninstall: protobuf
    Found existing installation: protobuf 3.18.1
    Uninstalling protobuf-3.18.1:
      Successfully uninstalled protobuf-3.18.1
Successfully installed cachetools-4.2.4 google-api-core-2.2.2 google-auth-2.3.3 google-cloud-bigquery-2.30.1 google-cloud-bigquery-storage-2.10.1 google-cloud-core-2.2.1 google-crc32c-1.3.0 google-resumable-media-2.1.0 googleapis-common-protos-1.53.0 grpcio-1.42.0 grpcio-status-1.42.0 libcst-0.3.23 mypy-extensions-0.4.3 proto-plus-1.19.8 protobuf-3.19.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 rsa-4.8 typing-inspect-0.7.1
In [ ]:
%load_ext google.cloud.bigquery
In [ ]:
%%bigquery df_example
SELECT * FROM (
  SELECT
    [414016,
    5767168,
    89019964] AS phone_numbers
  UNION ALL
  SELECT
    [8708227,
    37637318] AS phone_numbers
  UNION ALL
  SELECT
    [9349460,
    84826376,
    9215080,
    26331650] AS phone_numbers)
In [ ]:
df_example.to_dict()
Out[ ]:
{'phone_numbers': {0: [414016, 5767168, 89019964],
  1: [8708227, 37637318],
  2: [9349460, 84826376, 9215080, 26331650]}}
In [ ]:
phone_number_strings_list = []
for phone_numbers in df_example['phone_numbers']:
  phone_number_strings = []
  for phone_number in phone_numbers:
    phone_number_string = f'{phone_number:08d}'
    phone_number_strings.append(f'(+81) {phone_number_string[:4]}-{phone_number_string[-4:]}')
  phone_number_strings_list.append(phone_number_strings)
df_example['phone_number_strings'] = phone_number_strings_list
del df_example['phone_numbers']
In [ ]:
df_example.to_dict()
Out[ ]:
{'phone_number_strings': {0: ['(+81) 0041-4016',
   '(+81) 0576-7168',
   '(+81) 8901-9964'],
  1: ['(+81) 0870-8227', '(+81) 3763-7318'],
  2: ['(+81) 0934-9460',
   '(+81) 8482-6376',
   '(+81) 0921-5080',
   '(+81) 2633-1650']}}
In [ ]:
!sudo -HE pip install pandas_gbq

Collecting pandas_gbq
  Downloading https://files.pythonhosted.org/packages/c0/cb/d82930a55728359eb40e42e487d83315b09cb2ba316f0e889f5d4b886614/pandas_gbq-0.14.1-py3-none-any.whl
Requirement already satisfied: google-auth in /opt/tljh/user/lib/python3.6/site-packages (from pandas_gbq) (1.6.3)
Requirement already satisfied: pandas>=0.20.1 in /opt/tljh/user/lib/python3.6/site-packages (from pandas_gbq) (0.25.1)
Collecting google-auth-oauthlib (from pandas_gbq)
  Downloading https://files.pythonhosted.org/packages/b1/0e/0636cc1448a7abc444fb1b3a63655e294e0d2d49092dc3de05241be6d43c/google_auth_oauthlib-0.4.6-py2.py3-none-any.whl
Collecting pydata-google-auth (from pandas_gbq)
  Downloading https://files.pythonhosted.org/packages/d8/61/e4e0bae8906f3d2f460bc46c1ccd4a94caf7eaf65aa92421c48d7c56ef70/pydata_google_auth-1.2.0-py2.py3-none-any.whl
Requirement already satisfied: google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1 in /opt/tljh/user/lib/python3.6/site-packages (from pandas_gbq) (1.21.0)
Requirement already satisfied: setuptools in /opt/tljh/user/lib/python3.6/site-packages (from pandas_gbq) (41.4.0)
Requirement already satisfied: rsa>=3.1.4 in /opt/tljh/user/lib/python3.6/site-packages (from google-auth->pandas_gbq) (4.0)
Requirement already satisfied: cachetools>=2.0.0 in /opt/tljh/user/lib/python3.6/site-packages (from google-auth->pandas_gbq) (3.1.1)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/tljh/user/lib/python3.6/site-packages (from google-auth->pandas_gbq) (0.2.6)
Requirement already satisfied: six>=1.9.0 in /opt/tljh/user/lib/python3.6/site-packages (from google-auth->pandas_gbq) (1.12.0)
Requirement already satisfied: python-dateutil>=2.6.1 in /opt/tljh/user/lib/python3.6/site-packages (from pandas>=0.20.1->pandas_gbq) (2.8.0)
Requirement already satisfied: pytz>=2017.2 in /opt/tljh/user/lib/python3.6/site-packages (from pandas>=0.20.1->pandas_gbq) (2019.2)
Requirement already satisfied: numpy>=1.13.3 in /opt/tljh/user/lib/python3.6/site-packages (from pandas>=0.20.1->pandas_gbq) (1.17.2)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/tljh/user/lib/python3.6/site-packages (from google-auth-oauthlib->pandas_gbq) (1.2.0)
Requirement already satisfied: google-cloud-core<2.0dev,>=1.0.3 in /opt/tljh/user/lib/python3.6/site-packages (from google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1->pandas_gbq) (1.0.3)
Requirement already satisfied: protobuf>=3.6.0 in /opt/tljh/user/lib/python3.6/site-packages (from google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1->pandas_gbq) (3.13.0)
Requirement already satisfied: google-resumable-media!=0.4.0,<0.5.0dev,>=0.3.1 in /opt/tljh/user/lib/python3.6/site-packages (from google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1->pandas_gbq) (0.4.1)
Collecting google-cloud-bigquery-storage<2.0.0dev,>=0.6.0; extra == "bqstorage" (from google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1->pandas_gbq)
  Downloading https://files.pythonhosted.org/packages/42/9a/003822d79a535472c089ca39fb384b74b8a3624f4d5a1715c4c52059418d/google_cloud_bigquery_storage-1.1.0-py2.py3-none-any.whl (135kB)
     |████████████████████████████████| 143kB 37.3MB/s eta 0:00:01
Collecting pyarrow!=0.14.0,>=0.13.0; extra == "bqstorage" (from google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1->pandas_gbq)
  Downloading https://files.pythonhosted.org/packages/5d/61/4160ed11e0c149182baafc3a5bed1fb04395ba40705556d03c9244fb57d4/pyarrow-6.0.0-cp36-cp36m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (24.3MB)
     |████████████████████████████████| 24.4MB 43.3MB/s eta 0:00:01
Requirement already satisfied: pyasn1>=0.1.3 in /opt/tljh/user/lib/python3.6/site-packages (from rsa>=3.1.4->google-auth->pandas_gbq) (0.4.7)
Requirement already satisfied: requests>=2.0.0 in /opt/tljh/user/lib/python3.6/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib->pandas_gbq) (2.25.1)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/tljh/user/lib/python3.6/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib->pandas_gbq) (3.1.0)
Requirement already satisfied: google-api-core<2.0.0dev,>=1.14.0 in /opt/tljh/user/lib/python3.6/site-packages (from google-cloud-core<2.0dev,>=1.0.3->google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1->pandas_gbq) (1.14.2)
Requirement already satisfied: idna<3,>=2.5 in /opt/tljh/user/lib/python3.6/site-packages (from requests>=2.0.0->requests-oauthlib>=0.7.0->google-auth-oauthlib->pandas_gbq) (2.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/tljh/user/lib/python3.6/site-packages (from requests>=2.0.0->requests-oauthlib>=0.7.0->google-auth-oauthlib->pandas_gbq) (1.24.3)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/tljh/user/lib/python3.6/site-packages (from requests>=2.0.0->requests-oauthlib>=0.7.0->google-auth-oauthlib->pandas_gbq) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /opt/tljh/user/lib/python3.6/site-packages (from requests>=2.0.0->requests-oauthlib>=0.7.0->google-auth-oauthlib->pandas_gbq) (2020.4.5.2)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/tljh/user/lib/python3.6/site-packages (from google-api-core<2.0.0dev,>=1.14.0->google-cloud-core<2.0dev,>=1.0.3->google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1->pandas_gbq) (1.6.0)
Installing collected packages: google-auth-oauthlib, pydata-google-auth, pandas-gbq, google-cloud-bigquery-storage, pyarrow
Successfully installed google-auth-oauthlib-0.4.6 google-cloud-bigquery-storage-1.1.0 pandas-gbq-0.14.1 pyarrow-6.0.0 pydata-google-auth-1.2.0

https://console.cloud.google.com/bigquery?project=sandbox-sheng

In [ ]:
df_example.to_gbq(project_id = 'sandbox-sheng', destination_table = 'bigquery_learning.customers', if_exists = 'replace')

/home/jupyter-sheng_wei/.local/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
1it [00:04,  4.10s/it]
In [ ]:
%%bigquery
SELECT
  *
FROM
  `sandbox-sheng.bigquery_learning.customers`
Out[ ]:
phone_number_strings
0 ['(+81) 0041-4016', '(+81) 0576-7168', '(+81) ...
1 ['(+81) 0870-8227', '(+81) 3763-7318']
2 ['(+81) 0934-9460', '(+81) 8482-6376', '(+81) ...

IPython Magics for BigQuery

In [ ]:
from datetime import datetime, timezone, timedelta
# from 30 days ago to 29 days ago
start_date = datetime.now(timezone(timedelta(hours=+9))) - timedelta(days=30)
end_date = datetime.now(timezone(timedelta(hours=+9))) - timedelta(days=29)
params = {
    'start_date': start_date.strftime('%Y-%m-%d %H:%M:%S Asia/Tokyo'),
    'end_date': end_date.strftime('%Y-%m-%d %H:%M:%S Asia/Tokyo')
}
In [ ]:
print(params['start_date'])
print(params['end_date'])

2021-10-14 09:21:28 Asia/Tokyo
2021-10-15 09:21:28 Asia/Tokyo
In [ ]:
%%time
%%bigquery df --param $params
SELECT
  *
FROM
  `sandbox-sheng.fluentd.fluentd_test`
WHERE
  time >= @start_date
  AND time < @end_date
ORDER BY
  time ASC
In [ ]:
df

Read BigQuery Reference

In [ ]:
%%bigquery
SELECT
  ARRAY(
  SELECT
    CONCAT('(+81) ', SUBSTR(FORMAT('%08d', phone_number), 0, 4), '-', SUBSTR(FORMAT('%08d', phone_number), -4))
  FROM
    UNNEST(phone_numbers) AS phone_number )[OFFSET(0)] AS phone_number
FROM (
  SELECT
    [414016,
    5767168,
    89019964] AS phone_numbers
  UNION ALL
  SELECT
    [8708227,
    37637318] AS phone_numbers
  UNION ALL
  SELECT
    [9349460,
    84826376,
    9215080,
    26331650] AS phone_numbers)
Out[ ]:
phone_number
0 (+81) 0041-4016
1 (+81) 0870-8227
2 (+81) 0934-9460

https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#array_agg

In [ ]:
%%bigquery
WITH fruits AS
  (SELECT "apple" AS fruit
   UNION ALL SELECT "pear" AS fruit
   UNION ALL SELECT "banana" AS fruit)
SELECT ARRAY_AGG(fruit ORDER BY fruit) AS fruit_basket
FROM fruits
Out[ ]:
fruit_basket
0 [apple, banana, pear]
In [ ]:
%%bigquery
SELECT
  name,
  ARRAY_AGG( DISTINCT phone_number
  ORDER BY
    phone_number ASC )[OFFSET(0)] AS phone_number
FROM (
  SELECT
    'Andrew' AS name,
    414016 AS phone_number
  UNION ALL
  SELECT
    'Andrew' AS name,
    5767168 AS phone_number
  UNION ALL
  SELECT
    'Andrew' AS name,
    89019964 AS phone_number
  UNION ALL
  SELECT
    'Andrew' AS name,
    8708227 AS phone_number
  UNION ALL
  SELECT
    'Andrew' AS name,
    37637318 AS phone_number
  UNION ALL
  SELECT
    'Kelly' AS name,
    9349460 AS phone_number
  UNION ALL
  SELECT
    'Kelly' AS name,
    84826376 AS phone_number
  UNION ALL
  SELECT
    'Kelly' AS name,
    9215080 AS phone_number
  UNION ALL
  SELECT
    'Kelly' AS name,
    26331650 AS phone_number)
GROUP BY
  name
Out[ ]:
name phone_number
0 Andrew 414016
1 Kelly 9215080
Read More

2019-10-19
Try Jupyter Dashboards

Jupyter Dashboards

https://github.com/jupyter/dashboards

I think it is a bit like appmode.

https://github.com/oschuett/appmode

In [ ]:
!sudo -E pip install jupyter_dashboards
!sudo -E jupyter dashboards quick-setup --sys-prefix

Jupyter Dashboards Demos

In [ ]:
!sudo -HE pip install -r ~/dashboards.git/requirements-demo.txt

Notebooks can not be well shown in Grid Layout with my Jupyter Dashboards.

In [ ]:
!sudo -E jupyter dashboards quick-remove --sys-prefix
Read More

2019-10-19
Try Commuter

https://github.com/nteract/commuter

In [ ]:
!sudo -E npm install @nteract/commuter -g
In [ ]:
!sudo -E npm install --build-from-source canvas@2.6.0 -g --unsafe-perm=true --allow-root --verbose

Turn to yarn

In [ ]:
%%bash
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

sudo apt-get update && sudo apt-get install --no-install-recommends yarn
In [ ]:
!sudo -E yarn global add @nteract/commuter

Check Install Result

In [ ]:
!cat /usr/local/share/.config/yarn/global/node_modules/@nteract/commuter/package.json

Concerning commuter

In [ ]:
!cat /usr/local/share/.config/yarn/global/node_modules/@nteract/commuter/package.json | grep -1 --color commuter

{
  "name": "@nteract/commuter",
  "version": "5.9.0",
--
  "bin": {
    "commuter": "bin.js"
  },
--
  },
  "homepage": "https://github.com/nteract/nteract/tree/master/applications/commuter#readme",
  "publishConfig": {

Concerning bin.js

In [ ]:
!cat /usr/local/share/.config/yarn/global/node_modules/@nteract/commuter/bin.js

#!/usr/bin/env node

process.env.NODE_ENV = "production";

require("./lib/index.js");

Concerning backend

In [ ]:
!cat /usr/local/share/.config/yarn/global/node_modules/@nteract/commuter/package.json | grep -1 --color backend

    "build:frontend-for-production": "next build",
    "build:server": "babel backend -d lib",
    "build:all": "npm run build:server && npm run build:frontend-for-production",
In [ ]:
!sed -n '6,17p' /usr/local/share/.config/yarn/global/node_modules/@nteract/commuter/package.json

  "scripts": {
    "predev": "npm run build:server",
    "dev": "NODE_ENV=development node lib/index.js",
    "prestart": "npm run build:all",
    "start": "NODE_ENV=production node lib/index.js",
    "build": "npm run build:all",
    "build:frontend-for-production": "next build",
    "build:server": "babel backend -d lib",
    "build:all": "npm run build:server && npm run build:frontend-for-production",
    "prepublishOnly": "npm run build:all",
    "test": "jest"
  },

Concerning index.js

In [ ]:
!cat /usr/local/share/.config/yarn/global/node_modules/@nteract/commuter/lib/index.js

"use strict";

var log = require("log");

var createServer = require("./server");

createServer().then(function (server) {
  var port = server.address().port;
  console.log(log);
  log.info("Commuter server listening on port " + port);
})["catch"](function (e) {
  console.error(e);
  console.error(e.stack);
  process.exit(-10);
});

Concerning port

In [ ]:
!cat /usr/local/share/.config/yarn/global/node_modules/@nteract/commuter/pages/discover.js | grep -w1 --color port

      // Server side, communicate with our local API
      const port = process.env.COMMUTER_PORT || 4000;
      BASE_PATH = `http://127.0.0.1:${port}/`;
    } else {

Run Commuter

In [ ]:
!commuter
Read More

2019-10-13
Try PaperBoby

https://github.com/timkpaine/paperboy

In [ ]:
!cd ~ && git clone https://github.com/timkpaine/paperboy.git

Cloning into 'paperboy'...
remote: Enumerating objects: 21, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 3388 (delta 3), reused 3 (delta 0), pack-reused 3367
Receiving objects: 100% (3388/3388), 12.04 MiB | 22.38 MiB/s, done.
Resolving deltas: 100% (2277/2277), done.
In [ ]:
%%bash
cd ~/paperboy/
sudo -E npm install
sudo -E npm run build
sudo -E pip install -e .
In [ ]:
!sudo -E python -m paperboy

CRITICAL:root:Using SQL backend
CRITICAL:root:Using SQL auth
CRITICAL:root:Using dummy scheduler
[2019-10-13 18:36:29 +0900] [2696] [INFO] Starting gunicorn 19.9.0
[2019-10-13 18:36:29 +0900] [2696] [INFO] Listening at: http://0.0.0.0:8080 (2696)
[2019-10-13 18:36:29 +0900] [2696] [INFO] Using worker: sync
[2019-10-13 18:36:29 +0900] [2701] [INFO] Booting worker with pid: 2701
CRITICAL:root:Storing user sheng_wei eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6IjEiLCJuYW1lIjoic2hlbmdfd2VpIn0.oKJsYgZJaj1dLoVR8JgTjfPGp2O8bf3jqmoRux1bNgw 1
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 0 - 0
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 0 - 0
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
CRITICAL:root:Storing notebook <Notebook(name='TEST', user='<User(name='sheng_wei')>', privacy='public', level='production'>
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 0 - 0
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 0 - 0
[2019-10-13 18:43:59 +0900] [2701] [ERROR] Error handling request /api/v1/jobs?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/job.py", line 21, in on_post
    resp.body = json.dumps(self.db.jobs.store(req.context['user'], req.params, self.session, self.scheduler))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/job.py", line 42, in store
    nb_sql = session.query(NotebookSQL).get(int(justid(notebookid)))
ValueError: invalid literal for int() with base 10: 'attachment.ipynb'
[2019-10-13 18:44:22 +0900] [2701] [ERROR] Error handling request /api/v1/jobs?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/job.py", line 21, in on_post
    resp.body = json.dumps(self.db.jobs.store(req.context['user'], req.params, self.session, self.scheduler))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/job.py", line 42, in store
    nb_sql = session.query(NotebookSQL).get(int(justid(notebookid)))
ValueError: invalid literal for int() with base 10: 'attachment.ipynb'
[2019-10-13 18:44:23 +0900] [2701] [ERROR] Error handling request /api/v1/jobs?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/job.py", line 21, in on_post
    resp.body = json.dumps(self.db.jobs.store(req.context['user'], req.params, self.session, self.scheduler))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/job.py", line 42, in store
    nb_sql = session.query(NotebookSQL).get(int(justid(notebookid)))
ValueError: invalid literal for int() with base 10: 'attachment.ipynb'
[2019-10-13 18:44:23 +0900] [2701] [ERROR] Error handling request /api/v1/jobs?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/job.py", line 21, in on_post
    resp.body = json.dumps(self.db.jobs.store(req.context['user'], req.params, self.session, self.scheduler))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/job.py", line 42, in store
    nb_sql = session.query(NotebookSQL).get(int(justid(notebookid)))
ValueError: invalid literal for int() with base 10: 'attachment.ipynb'
[2019-10-13 18:44:23 +0900] [2701] [ERROR] Error handling request /api/v1/jobs?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/job.py", line 21, in on_post
    resp.body = json.dumps(self.db.jobs.store(req.context['user'], req.params, self.session, self.scheduler))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/job.py", line 42, in store
    nb_sql = session.query(NotebookSQL).get(int(justid(notebookid)))
ValueError: invalid literal for int() with base 10: 'attachment.ipynb'
CRITICAL:root:Storing job <Job(name='TEST', user='<User(name='sheng_wei')>', notebook='<Notebook(name='TEST', user='<User(name='sheng_wei')>', privacy='public', level='production'>', start='2019-10-13 09:37:00', interval='minutely'>
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 1 - 1
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 1 - 1
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 1 - 1
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
CRITICAL:root:detail : 1, result : <Job(name='TEST', user='<User(name='sheng_wei')>', notebook='<Notebook(name='TEST', user='<User(name='sheng_wei')>', privacy='public', level='production'>', start='2019-10-13 09:37:00', interval='minutely'>
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 1 - 1
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 1 - 1
CRITICAL:root:detail : 1, result : <Job(name='TEST', user='<User(name='sheng_wei')>', notebook='<Notebook(name='TEST', user='<User(name='sheng_wei')>', privacy='public', level='production'>', start='2019-10-13 09:37:00', interval='minutely'>
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 1 - 1
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 1 - 1
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
CRITICAL:root:detail : 1, result : <Notebook(name='TEST', user='<User(name='sheng_wei')>', privacy='public', level='production'>
CRITICAL:root:Storing notebook <Notebook(name='BQ', user='<User(name='sheng_wei')>', privacy='public', level='production'>
CRITICAL:root:detail : 2, result : <Notebook(name='BQ', user='<User(name='sheng_wei')>', privacy='public', level='production'>
[2019-10-13 18:54:20 +0900] [2701] [ERROR] Error handling request /api/v1/jobs?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/job.py", line 21, in on_post
    resp.body = json.dumps(self.db.jobs.store(req.context['user'], req.params, self.session, self.scheduler))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/job.py", line 42, in store
    nb_sql = session.query(NotebookSQL).get(int(justid(notebookid)))
ValueError: invalid literal for int() with base 10: 'BQ'
[2019-10-13 18:54:35 +0900] [2701] [ERROR] Error handling request /api/v1/jobs?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/job.py", line 21, in on_post
    resp.body = json.dumps(self.db.jobs.store(req.context['user'], req.params, self.session, self.scheduler))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/job.py", line 42, in store
    nb_sql = session.query(NotebookSQL).get(int(justid(notebookid)))
ValueError: invalid literal for int() with base 10: 'BQ'
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 2 - 2
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 2 - 2
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 2 - 2
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 2 - 2
CRITICAL:root:detail : 2, result : <Notebook(name='BQ', user='<User(name='sheng_wei')>', privacy='public', level='production'>
CRITICAL:root:Storing job <Job(name='BQ', user='<User(name='sheng_wei')>', notebook='<Notebook(name='BQ', user='<User(name='sheng_wei')>', privacy='public', level='production'>', start='2019-10-13 09:51:00', interval='minutely'>
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 2 - 2
CRITICAL:root:detail : 2, result : <Job(name='BQ', user='<User(name='sheng_wei')>', notebook='<Notebook(name='BQ', user='<User(name='sheng_wei')>', privacy='public', level='production'>', start='2019-10-13 09:51:00', interval='minutely'>
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
[2019-10-13 18:59:11 +0900] [2696] [CRITICAL] WORKER TIMEOUT (pid:2701)
[2019-10-13 18:59:11 +0900] [2701] [INFO] Worker exiting (pid: 2701)
[2019-10-13 18:59:11 +0900] [2863] [INFO] Booting worker with pid: 2863
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 2 - 2
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 2 - 2
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
CRITICAL:root:detail : 2, result : <Job(name='BQ', user='<User(name='sheng_wei')>', notebook='<Notebook(name='BQ', user='<User(name='sheng_wei')>', privacy='public', level='production'>', start='2019-10-13 09:51:00', interval='minutely'>
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 2 - 2
[2019-10-13 19:01:55 +0900] [2863] [ERROR] Error handling request /api/v1/reports?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/report.py", line 21, in on_post
    resp.body = json.dumps(self.db.reports.store(req.context['user'], req.params, self.session))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/report.py", line 50, in store
    jb_sql = session.query(JobSQL).get(int(justid(jobid)))
ValueError: invalid literal for int() with base 10: 'BQ'
[2019-10-13 19:01:57 +0900] [2863] [ERROR] Error handling request /api/v1/reports?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/report.py", line 21, in on_post
    resp.body = json.dumps(self.db.reports.store(req.context['user'], req.params, self.session))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/report.py", line 50, in store
    jb_sql = session.query(JobSQL).get(int(justid(jobid)))
ValueError: invalid literal for int() with base 10: 'BQ'
[2019-10-13 19:01:57 +0900] [2863] [ERROR] Error handling request /api/v1/reports?action=save
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/tljh/user/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/tljh/user/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__
    responder(req, resp, **params)
  File "/home/jupyter-sheng_wei/paperboy/paperboy/resources/report.py", line 21, in on_post
    resp.body = json.dumps(self.db.reports.store(req.context['user'], req.params, self.session))
  File "/home/jupyter-sheng_wei/paperboy/paperboy/storage/sqla/report.py", line 50, in store
    jb_sql = session.query(JobSQL).get(int(justid(jobid)))
ValueError: invalid literal for int() with base 10: 'BQ'
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.notebook.NotebookSQL'>, result : 2 - 2
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.job.JobSQL'>, result : 2 - 2
CRITICAL:root:list : <class 'paperboy.storage.sqla.models.report.ReportSQL'>, result : 0 - 0
^C
[2019-10-13 19:02:42 +0900] [2696] [INFO] Handling signal: int
[2019-10-13 19:02:42 +0900] [2863] [INFO] Worker exiting (pid: 2863)

The versions of dependencies are too low. There are too many bugs.

It seems that the maintenance of PaperBoy has been stopped.

Well. Maybe I can reuse some of the codes.

But I found a new Jupyter Dashboard used by PaperBoy.

Read More

2019-10-08
Try Appmod and NBGitPuller

https://mybinder.org/v2/gh/jupyterhub/nbgitpuller/master?urlpath=apps/binder%2Flink_generator.ipynb

Try Appmode

In [ ]:
!sudo -E pip install appmode
!sudo -E jupyter nbextension     enable --py --sys-prefix appmode
!sudo -E jupyter serverextension enable --py --sys-prefix appmode

Try NBGitPuller

In [ ]:
!sudo -E pip install -U nbgitpuller
In [ ]:
from nbgitpuller import pull
pull.__file__
In [ ]:
from nbgitpuller import handlers
handlers.__file__

Change repo_url to clone_cmd

In [ ]:
!cd ~ && git clone --branch change-to-clone-command https://github.com/seii-saintway/nbgitpuller.git
In [ ]:
!sudo cp ~/nbgitpuller/nbgitpuller/pull.py /opt/tljh/user/lib/python3.6/site-packages/nbgitpuller/pull.py
!sudo cp ~/nbgitpuller/nbgitpuller/handlers.py /opt/tljh/user/lib/python3.6/site-packages/nbgitpuller/handlers.py
  • https://jhub.name
  • git clone https://github.com/seii-saintway/nbgitpuller
  • change-to-clone-command
  • binder/link_generator.ipynb
  • app

https://jhub.name/hub/user-redirect/git-pull?repo=git+clone+https%3A%2F%2Fgithub.com%2Fseii-saintway%2Fnbgitpuller&branch=change-to-clone-command&subPath=binder%2Flink_generator.ipynb&app=app

Installation

In [ ]:
!sudo -E pip install git+https://github.com/seii-saintway/nbgitpuller@change-to-clone-command
Read More

2019-10-06
Make isso HTTPS with TLJH Traefik Proxy

Make Isso HTTPS in TLJH

In [ ]:
%%bash
/opt/tljh/hub/bin/traefik version

for traefik v1.7

In [ ]:
%%bash
cat << EOF | sudo tee -a /opt/tljh/state/rules.toml > /dev/null

[frontends.isso]
backend = "isso"
passHostHeader = true

[frontends.isso.routes.f1]
rule = "PathPrefixStrip:/isso/"

[backends.isso.servers.s1]
url = "http://127.0.0.1:8000"
weight = 1
EOF
In [ ]:
%env http=http:\/\/jhub.name:8000\/
%env https=https:\/\/jhub.name\/isso\/
!sed -i -e "s/$http/$https/g" ~/hexo/themes/freemind/layout/_partial/post/comment.ejs

for traefik v2.10 StripPrefix

In [ ]:
%%bash
cat << EOF | sudo tee /opt/tljh/state/rules/isso.toml > /dev/null
[http.routers.isso]
service = "isso"
rule = "PathPrefix(\`/isso/\`)"
entryPoints = [ "https",]
middlewares = ["isso"]

[http.middlewares]
  [http.middlewares.isso.stripPrefix]
    prefixes = ["/isso"]

[http.services.isso.loadBalancer]
passHostHeader = true
[[http.services.isso.loadBalancer.servers]]
url = "http://127.0.0.1:8000"
EOF

Commit Modifications

https://github.com/seii-saintway/hexo-theme-freemind/commit/9568c3160d3c917a8a986beedafb7836e02c2bce

Change file-watch to directory in TLJH Traefik

https://github.com/seii-saintway/the-littlest-jupyterhub/commit/e95d85e99187f9d65b748016807b28c256e77e39

In [ ]:
%%bash
cat << EOF | sudo tee /opt/tljh/state/dynamic/isso.toml > /dev/null
[frontends.isso]
backend = "isso"
passHostHeader = true

[frontends.isso.routes.f1]
rule = "PathPrefixStrip:/isso/"

[backends.isso.servers.s1]
url = "http://127.0.0.1:8000"
weight = 1
EOF
In [ ]:
!sudo /opt/tljh/hub/bin/python -m tljh.installer

Granting passwordless sudo to JupyterHub admins...
Setting up user environment...


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.8
  latest version: 4.7.12

Please update conda by running

    $ conda update -n base conda


Setting up JupyterHub...
In [ ]:
!sudo cat /opt/tljh/installer.log

2019-10-13 15:21:02,889 Granting passwordless sudo to JupyterHub admins...
2019-10-13 15:21:02,890 Setting up user environment...
2019-10-13 15:21:26,052 Setting up JupyterHub...
In [ ]:
!cd ~ && make restart-isso

Isso SQLite

In [ ]:
%%bash
cat << EOF | sudo tee /etc/isso.conf > /dev/null

[general]

dbpath = /var/lib/isso/comments.db

host =
    http://seii-saintway.github.io
    https://seii-saintway.github.io
    http://digitypography.xyz
    https://digitypography.xyz

EOF

Open SQLite database

sqlite3 /var/lib/isso/comments.db
In [ ]:
import sqlite3
conn = sqlite3.connect('/var/lib/isso/comments.db')
In [ ]:
[(row[6], row[7]) for row in conn.cursor().execute('SELECT * from comments')]
Out[ ]:
[('110.3.247.0', '# Privacy Policy'),
 ('113.38.215.0', '# Continuous Purification'),
 ('110.3.247.0',
  '### 3.11.4 隐秘通道\n\n隐秘通道（covert channel）的意思是说使用难以预料或不会引起注意的方式来发送信息。假设有一天系统管理员Sally觉得自己的那些用户花太多时间来玩了，因此她决定关闭email和即时消息，这样他们就不能聊天了。要解决这个问题，你和你的朋友都同意使用主目录中的一个你们都可以读取的文件来交流信息，你每隔一会儿就得检查一下该文件来查看新消息。这种超出乎预料的通信方式就是一个隐秘通道。\n\n隐秘通道很难取消。如果系统管理员Sally发现了你们这种基于文件的通信技术，她可以修改目录的权限，使得只有目录的属主才具有读取和搜索的权限，并限定目录属主也不能修改这种权限。在这样做的同时，她还可以确认你不能在其他地方（例如，/tmp）创建文件。（你的大部分程序都不能运行了，但是不要因此而责怪Sally。）即使是这样，你和你的朋友仍然可以相互显示其他用户士目录中的内容，这可以显示目录的修改日期和文件个数，这样你就可以根据这些可见的参数设计一种密码，并通过修改这些参数而进行通信。这是一种更复杂的隐秘通道，如果Sally对你们进行了更严格的限制，你们可以想出更古怪的法子来。\n\nSSH不能防止隐秘通道。对隐秘通道的分析和控制通常都是安全性很高的计算机系统的一部分，例如设计用来在同一个系统中的不同安全层次上安全地处理信息的系统。顺便说一下，SSH数据流本身就可以很好地用作一种隐秘通道：SSH会话的内容可能是一个巧克力馅饼的配方，而隐含的内容则可能是用Morse码表示这两个公司即将合并，只需要使用报文长度是奇数还是偶数来分别表示Morse码的短横线和点号即可。'),
 ('126.158.192.0', 'Try to build with Cosmos SDK'),
 ('126.158.192.0', 'Try to build with Cosmos SDK')]
In [ ]:
[row for row in conn.cursor().execute('SELECT * from preferences')]
In [ ]:
[row for row in conn.cursor().execute('SELECT * from threads')]
Out[ ]:
[(1, '/2019/07/17/%E5%BF%83%E6%83%85%E7%B3%9F%E9%80%8F%E4%BA%86/', None),
 (2, '/2022/02/25/Declaration-of-the-Independence-of-Cyberspace/', None),
 (3, '/2022/02/28/An-Independent-Stablecoin-of-Cyberspace/', None)]
In [ ]:
conn.close()
Read More

2019-10-06
Try To Import Notebooks

In [ ]:
import Try_To_Import_Python

Try_To_Import_Python.py

https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Importing%20Notebooks.html

Here is some simple code to display the contents of a notebook with syntax highlighting, etc.

In [ ]:
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

from IPython.display import display, HTML

formatter = HtmlFormatter()
lexer = PythonLexer()

# publish the CSS for pygments highlighting
display(HTML("""
<style type='text/css'>
%s
</style>
""" % formatter.get_style_defs()
))
In [ ]:
def show_notebook(fname):
    """display a short summary of the cells of a notebook"""
    import io
    with io.open(fname, 'r', encoding='utf-8') as f:
        from nbformat import read
        nb = read(f, 4)
    html = []
    for cell in nb.cells:
        html.append("<h4>%s cell</h4>" % cell.cell_type)
        if cell.cell_type == 'code':
            html.append(highlight(cell.source, lexer, formatter))
        else:
            html.append("<pre>%s</pre>" % cell.source)
    display(HTML('\n'.join(html)))

show_notebook('Importing Notebooks.ipynb')

markdown cell

# Importing Jupyter Notebooks as Modules

markdown cell

It is a common problem that people want to import code from Jupyter Notebooks.
This is made difficult by the fact that Notebooks are not plain Python files,
and thus cannot be imported by the regular Python machinery.

Fortunately, Python provides some fairly sophisticated [hooks](https://www.python.org/dev/peps/pep-0302/) into the import machinery,
so we can actually make Jupyter notebooks importable without much difficulty,
and only using public APIs.

code cell

import io, os, sys, types

code cell

from IPython import get_ipython
from nbformat import read
from IPython.core.interactiveshell import InteractiveShell

markdown cell

Import hooks typically take the form of two objects:

1. a Module **Loader**, which takes a module name (e.g. `'IPython.display'`), and returns a Module
2. a Module **Finder**, which figures out whether a module might exist, and tells Python what **Loader** to use

code cell

def find_notebook(fullname, path=None):
    """find a notebook, given its fully qualified name and an optional path
    
    This turns "foo.bar" into "foo/bar.ipynb"
    and tries turning "Foo_Bar" into "Foo Bar" if Foo_Bar
    does not exist.
    """
    name = fullname.rsplit('.', 1)[-1]
    if not path:
        path = ['']
    for d in path:
        nb_path = os.path.join(d, name + ".ipynb")
        if os.path.isfile(nb_path):
            return nb_path
        # let import Notebook_Name find "Notebook Name.ipynb"
        nb_path = nb_path.replace("_", " ")
        if os.path.isfile(nb_path):
            return nb_path

markdown cell

## Notebook Loader

markdown cell

Here we have our Notebook Loader.
It's actually quite simple - once we figure out the filename of the module,
all it does is:

1. load the notebook document into memory
2. create an empty Module
3. execute every cell in the Module namespace

Since IPython cells can have extended syntax,
the IPython transform is applied to turn each of these cells into their pure-Python counterparts before executing them.
If all of your notebook cells are pure-Python,
this step is unnecessary.

code cell

class NotebookLoader(object):
    """Module Loader for Jupyter Notebooks"""
    def __init__(self, path=None):
        self.shell = InteractiveShell.instance()
        self.path = path
    
    def load_module(self, fullname):
        """import a notebook as a module"""
        path = find_notebook(fullname, self.path)
        
        print ("importing Jupyter notebook from %s" % path)
                                       
        # load the notebook object
        with io.open(path, 'r', encoding='utf-8') as f:
            nb = read(f, 4)
        
        
        # create the module and add it to sys.modules
        # if name in sys.modules:
        #    return sys.modules[name]
        mod = types.ModuleType(fullname)
        mod.__file__ = path
        mod.__loader__ = self
        mod.__dict__['get_ipython'] = get_ipython
        sys.modules[fullname] = mod
        
        # extra work to ensure that magics that would affect the user_ns
        # actually affect the notebook module's ns
        save_user_ns = self.shell.user_ns
        self.shell.user_ns = mod.__dict__
        
        try:
          for cell in nb.cells:
            if cell.cell_type == 'code':
                # transform the input to executable Python
                code = self.shell.input_transformer_manager.transform_cell(cell.source)
                # run the code in themodule
                exec(code, mod.__dict__)
        finally:
            self.shell.user_ns = save_user_ns
        return mod

markdown cell

## The Module Finder

markdown cell

The finder is a simple object that tells you whether a name can be imported,
and returns the appropriate loader.
All this one does is check, when you do:

```python
import mynotebook
```

it checks whether `mynotebook.ipynb` exists.
If a notebook is found, then it returns a NotebookLoader.

Any extra logic is just for resolving paths within packages.

code cell

class NotebookFinder(object):
    """Module finder that locates Jupyter Notebooks"""
    def __init__(self):
        self.loaders = {}
    
    def find_module(self, fullname, path=None):
        nb_path = find_notebook(fullname, path)
        if not nb_path:
            return
        
        key = path
        if path:
            # lists aren't hashable
            key = os.path.sep.join(path)
        
        if key not in self.loaders:
            self.loaders[key] = NotebookLoader(path)
        return self.loaders[key]

markdown cell

## Register the hook

markdown cell

Now we register the `NotebookFinder` with `sys.meta_path`

code cell

sys.meta_path.append(NotebookFinder())

markdown cell

After this point, my notebooks should be importable.

Let's look at what we have in the CWD:

code cell

ls nbpackage

markdown cell

So I should be able to `import nbpackage.mynotebook`.

code cell

import nbpackage.mynotebook

markdown cell

### Aside: displaying notebooks

markdown cell

Here is some simple code to display the contents of a notebook
with syntax highlighting, etc.

code cell

from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

from IPython.display import display, HTML

formatter = HtmlFormatter()
lexer = PythonLexer()

# publish the CSS for pygments highlighting
display(HTML("""
<style type='text/css'>
%s
</style>
""" % formatter.get_style_defs()
))

code cell

def show_notebook(fname):
    """display a short summary of the cells of a notebook"""
    with io.open(fname, 'r', encoding='utf-8') as f:
        nb = read(f, 4)
    html = []
    for cell in nb.cells:
        html.append("<h4>%s cell</h4>" % cell.cell_type)
        if cell.cell_type == 'code':
            html.append(highlight(cell.source, lexer, formatter))
        else:
            html.append("<pre>%s</pre>" % cell.source)
    display(HTML('\n'.join(html)))

show_notebook(os.path.join("nbpackage", "mynotebook.ipynb"))

markdown cell

So my notebook has some code cells,
one of which contains some IPython syntax.

Let's see what happens when we import it

code cell

from nbpackage import mynotebook

markdown cell

Hooray, it imported!  Does it work?

code cell

mynotebook.foo()

markdown cell

Hooray again!

Even the function that contains IPython syntax works:

code cell

mynotebook.has_ip_syntax()

code cell

mynotebook.whatsmyname()

markdown cell

## Notebooks in packages

markdown cell

We also have a notebook inside the `nb` package,
so let's make sure that works as well.

code cell

ls nbpackage/nbs

markdown cell

Note that the `__init__.py` is necessary for `nb` to be considered a package,
just like usual.

code cell

show_notebook(os.path.join("nbpackage", "nbs", "other.ipynb"))

code cell

from nbpackage.nbs import other
other.bar(5)

markdown cell

So now we have importable notebooks, from both the local directory and inside packages.

I can even put a notebook inside IPython, to further demonstrate that this is working properly:

code cell

import shutil
from IPython.paths import get_ipython_package_dir

utils = os.path.join(get_ipython_package_dir(), 'utils')
shutil.copy(os.path.join("nbpackage", "mynotebook.ipynb"),
            os.path.join(utils, "inside_ipython.ipynb")
)

markdown cell

and import the notebook from `IPython.utils`

code cell

from IPython.utils import inside_ipython
inside_ipython.whatsmyname()

markdown cell

This approach can even import functions and classes that are defined in a notebook using the `%%cython` magic.

A simple way to import .ipynb files in jupyter notebook are as follows:

  1. Install import-ipynb
In [ ]:
!sudo -E pip install import-ipynb
  1. Import import_ipynb in jupyter notebook. Then import .ipynb file as you import .py file.
In [ ]:
import import_ipynb
from nbpackage import mynotebook

mynotebook.whatsmyname()

importing Jupyter notebook from /home/jupyter-sheng_wei/hexo/source/notebooks/Try-Notebook-Import/nbpackage/mynotebook.ipynb
Out[ ]:
'nbpackage.mynotebook'
In [ ]:
mynotebook.__name__
Out[ ]:
'nbpackage.mynotebook'
In [ ]:
mynotebook.__file__
Out[ ]:
'/home/jupyter-sheng_wei/hexo/source/notebooks/Try-Notebook-Import/nbpackage/mynotebook.ipynb'
In [ ]:
import import_ipynb
from nbpackage.nbs import other
other.bar(10)

importing Jupyter notebook from /home/jupyter-sheng_wei/hexo/source/notebooks/Try-Notebook-Import/nbpackage/nbs/other.ipynb
Out[ ]:
'barbarbarbarbarbarbarbarbarbar'
Read More

2019-10-06
Try Datalab

try Datalab

launching Datalab on the Google Cloud Platform

https://cloud.google.com/datalab/docs/quickstart

launching Datalab locally

https://github.com/googledatalab/datalab/wiki/Getting-Started

In [ ]:
!git clone https://github.com/GoogleCloudPlatform/datalab.git
In [ ]:
!PROJECT_ID=sandbox-243014 datalab/containers/datalab/build.sh
In [ ]:
!PROJECT_ID=sandbox-243014 datalab/containers/datalab/run.sh
In [ ]:
!docker run --rm -it -p "8081:8080" -v "${HOME}:/content" -e "PROJECT_ID=sandbox-243014" gcr.io/cloud-datalab/datalab:local

try pydatalab

In [ ]:
!sudo -E pip install datalab
!sudo -E jupyter nbextension install --py datalab.notebook --sys-prefix

!sudo -E pip install google.cloud.monitoring
In [ ]:
!jupyter nbextension list
In [ ]:
%load_ext google.datalab.kernel
%gcs read --object gs://fluentd_test/avro --variable t
print(t)

samples of using pydatalab

https://github.com/googledatalab/notebooks

In [ ]:
!git clone https://github.com/googledatalab/notebooks.git
In [ ]:
!ls 'notebooks/tutorials/BigQuery/Importing and Exporting Data.ipynb'
!ls 'notebooks/tutorials/Storage/Storage Commands.ipynb'
In [ ]:
?set_datalab_project_id
In [ ]:
%load_ext google.datalab.kernel
%gcs list --objects gs://fluentd_test
In [ ]:
%load_ext google.datalab.kernel
In [ ]:
%gcs list --project sandbox-243014 --objects gs://fluentd_test
In [ ]:
%env PROJECT_ID=sandbox-243014

https://cloud.google.com/bigquery/docs/datalab-migration

source control

https://cloud.google.com/datalab/docs/how-to/working-with-notebooks

Read More

2019-10-06
Solve PyDatalab Incompatible with Python 3.7

https://github.com/googledatalab/pydatalab/issues/711

In [ ]:
!git clone https://github.com/googledatalab/pydatalab.git
In [ ]:
!grep -r 'async' --include='*.py' pydatalab/datalab/

https://github.com/googledatalab/pydatalab/pull/720/files

In [ ]:
!pip install --upgrade requests grpcio
!cd pydatalab && pip install .
In [ ]:
!grep -r 'async' --include='*.py' /home/sheng_wei/anaconda3/lib/python3.7/site-packages/datalab/
In [ ]:
!grep -r 'yaml' --include='*.py' /home/sheng_wei/anaconda3/lib/python3.7/site-packages/google/datalab/utils/commands/_utils.py
Read More

2019-10-05
Hexo Generate Notebook CSS

Remove Duplicated Notebook CSS

  • Add Category and Tag to Notebooks
  • Move Notebook CSS to <head></head>

Add Category and Tag to Notebooks

I find that the hexo layouts for Categories and Tags are almost the same, but it seems that the the usage of Tags is more casual. So I decide to add Tag "Notebooks" to my Notebook posts.

Move Notebook CSS to <head></head>

Read More
▲