Page MenuHomePhabricator
Create Task
Maniphest T273789

Sanitize and ingest all event tables into the event_sanitized database
Closed, ResolvedPublic
Actions

Description

We are slowly creating more streams in Event Platform, but we don't currently sanitize and save them in the event_sanitized database like we do for legacy EventLogging data.

For example: mediawiki_mediasearch_interaction

Acceptance criteria:

  • Allowlist system – file(s) where users can specify retention policies for streams
  • Sanitization job that sanitizes event data from Event Platform streams according to retention policies specified with the allowlist system

TODO:

  • Refactor EventLoggingSanitization job to something more generic: RefineSanitize
  • Move all event_sanitized partitions to lowercased directory names to avoid re-refining data
  • Apply backwards compatible usage of RefineSanitize for eventlogging
  • Create new generic RefineSanitize job to be able to sanitize any data
  • Copy all existent data that generic RefineSanitize job targets into event_sanitized
  • Create data purge job to remove data after 90 days from all tables in event

Details

SubjectRepoBranchLines +/-
refine: rename EventLoggingSanitization to RefineSanitizeoperations/puppetproduction+4 -4
data_purge - Refactor drop-el-unsanitized-events in to drop_event joboperations/puppetproduction+16 -2
analytics:refinery:job:data_purge: improve drop-el-unsanitized-eventsoperations/puppetproduction+1 -1
analytics:refinery:job:test:data_purge: remove -skipTrash from drop_eventoperations/puppetproduction+1 -1
Remove absented druid data drop job from test clusteroperations/puppetproduction+0 -13
Remove refinery-drop-webrequest-sampled-druid job from test clusteroperations/puppetproduction+2 -0
Add comment in refine about mediawiki_page_properties_changeoperations/puppetproduction+1 -0
Enable event_sanitized_main_immediate joboperations/puppetproduction+23 -24
Add comment in allowlist about mediawiki_page_properties_changeanalytics/refinerymaster+1 -1
Update event_sanitized_main_allowlistanalytics/refinerymaster+3 -2
canary_events and refine_sanitize - use refinery 0.1.9operations/puppetproduction+3 -3
Update refine_sanitize job to use event_sanitized_analytics_allowlistoperations/puppetproduction+2 -4
RefineSanitizeMonitor - pass keep_all_enabled to SanitizeTransformation.loadAllowlistanalytics/refinery/sourcemaster+3 -2
test/data_purge - add drop_event joboperations/puppetproduction+14 -1
Add more tables to sanitize in event_sanitized_main_allowlistanalytics/refinerymaster+16 -0
Rename and symlink sanitization eventlogging/whitelist.yamlanalytics/refinerymaster+21 -0
refine - allow configuring RefineMonitors since and until paramsoperations/puppetproduction+51 -24
refine_sanitize - use refinery 0.1.6 with RefineSanitize joboperations/puppetproduction+13 -64
sanitize_eventlogging_analytics_immediate - ensure absent during switchoveroperations/puppetproduction+6 -2
Move sanitize_eventlogging_analytics jobs from data_purge to refine_santizieoperations/puppetproduction+184 -100
test/refine_sanitized - add a event_sanitized_main joboperations/puppetproduction+69 -30
event_sanitized_main_allowlist - mediawiki_page_delete: keep_allanalytics/refinerymaster+1 -0
test/refine_sanitized - add a general purpose event_sanitized joboperations/puppetproduction+42 -4
SanitizeTransformation - Just some simple logging improvements.analytics/refinery/sourcemaster+12 -5
analytics test - import mediawiki.page-delete with camus and refine itoperations/puppetproduction+21 -0
test/refine_sanitize - absent sanitize delayed_test until Juneoperations/puppetproduction+3 -0
refine_sanitize_salt_rotate - use local_salts_prefix properlyoperations/puppetproduction+1 -1
test/refine_sanitize - make salt rotation more genericoperations/puppetproduction+149 -45
refine - use 0.1.5 and lowercase table names in include/exclude regexesoperations/puppetproduction+5 -5
test/refine - use 0.1.5 and lowercase table regexesoperations/puppetproduction+2 -2
test/refine_santize - Use normalized lowercase table nameoperations/puppetproduction+1 -1
test/refine_sanitize - ensure hdfs salts dir existsoperations/puppetproduction+12 -1
Ensure refinery/python on PYTHONPATH for refinery-eventlogging-saltrotateoperations/puppetproduction+5 -0
test/refine_sanitize - use proper refinery pathoperations/puppetproduction+2 -2
Set up refine_sanitize jobs in analytics test cluster.operations/puppetproduction+96 -0
Refine - fix typo in job_configoperations/puppetproduction+1 -1
refine - use refinery 0.1.4operations/puppetproduction+20 -20
refine_job - remove RefineFailuresChecker and use 0.1.4 in test/refineoperations/puppetproduction+84 -126
Use versioned refinery-job.jar in refine sanitize joboperations/puppetproduction+10 -0
Refactor EventLoggingSanitization to a generic job: RefineSanitizeanalytics/refinery/sourcemaster+487 -523
Add support for finding RefineTarget inputs from Hiveanalytics/refinery/sourcemaster+1 K -805
Rename whitelist to allowlist for Refine sanitizationanalytics/refinery/sourcemaster+194 -175
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Apr 13 2021, 1:17 PM

Change 678608 merged by Ottomata:

[operations/puppet@production] refine - use refinery 0.1.4

https://gerrit.wikimedia.org/r/678608

gerritbot added a comment.Apr 13 2021, 1:23 PM

Change 678836 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Refine - fix typo in job_config

https://gerrit.wikimedia.org/r/678836

gerritbot added a comment.Apr 13 2021, 1:24 PM

Change 678836 merged by Ottomata:

[operations/puppet@production] Refine - fix typo in job_config

https://gerrit.wikimedia.org/r/678836

gerritbot added a comment.Apr 13 2021, 2:37 PM

Change 676380 merged by Ottomata:

[operations/puppet@production] Set up refine_sanitize jobs in analytics test cluster.

https://gerrit.wikimedia.org/r/676380

gerritbot added a comment.Apr 13 2021, 2:42 PM

Change 678857 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/refine_sanitize - use proper refinery path

https://gerrit.wikimedia.org/r/678857

gerritbot added a comment.Apr 13 2021, 2:46 PM

Change 678857 merged by Ottomata:

[operations/puppet@production] test/refine_sanitize - use proper refinery path

https://gerrit.wikimedia.org/r/678857

gerritbot added a comment.Apr 13 2021, 2:53 PM

Change 678863 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Ensure refinery/python on PYTHONPATH for refinery-eventlogging-saltrotate in test

https://gerrit.wikimedia.org/r/678863

gerritbot added a comment.Apr 13 2021, 2:57 PM

Change 678863 merged by Ottomata:

[operations/puppet@production] Ensure refinery/python on PYTHONPATH for refinery-eventlogging-saltrotate

https://gerrit.wikimedia.org/r/678863

gerritbot added a comment.Apr 13 2021, 3:08 PM

Change 678867 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/refine_sanitize - ensure hdfs salts dir exists, and use -f when running saltrotate rm

https://gerrit.wikimedia.org/r/678867

gerritbot added a comment.Apr 13 2021, 3:11 PM

Change 678867 merged by Ottomata:

[operations/puppet@production] test/refine_sanitize - ensure hdfs salts dir exists

https://gerrit.wikimedia.org/r/678867

gerritbot added a comment.Apr 13 2021, 3:36 PM

Change 678876 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/refine_santize - Use normalized lowercase table name

https://gerrit.wikimedia.org/r/678876

gerritbot added a comment.Apr 13 2021, 3:40 PM

Change 678876 merged by Ottomata:

[operations/puppet@production] test/refine_santize - Use normalized lowercase table name

https://gerrit.wikimedia.org/r/678876

gerritbot added a comment.Apr 13 2021, 8:19 PM

Change 678941 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Refactor EventLoggingSanitization using RefineSanitize

https://gerrit.wikimedia.org/r/678941

gerritbot added a comment.Apr 14 2021, 4:00 PM

Change 679376 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] refine - lowercase eventlogging legeacy table names in include/exclude regexes

https://gerrit.wikimedia.org/r/679376

gerritbot added a comment.Apr 15 2021, 3:15 PM

Change 679846 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/refine - use 0.1.5 and lowercase table regexes

https://gerrit.wikimedia.org/r/679846

gerritbot added a comment.Apr 15 2021, 3:16 PM

Change 679846 merged by Ottomata:

[operations/puppet@production] test/refine - use 0.1.5 and lowercase table regexes

https://gerrit.wikimedia.org/r/679846

gerritbot added a comment.Apr 15 2021, 4:38 PM

Change 679376 merged by Ottomata:

[operations/puppet@production] refine - use 0.1.5 and lowercase table names in include/exclude regexes

https://gerrit.wikimedia.org/r/679376

gerritbot added a comment.Apr 15 2021, 8:15 PM

Change 679939 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/refine_sanitize - make salt rotation more generic

https://gerrit.wikimedia.org/r/679939

gerritbot added a comment.Apr 15 2021, 8:18 PM

Change 679939 merged by Ottomata:

[operations/puppet@production] test/refine_sanitize - make salt rotation more generic

https://gerrit.wikimedia.org/r/679939

gerritbot added a comment.Apr 15 2021, 8:23 PM

Change 679941 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] refine_sanitize_salt_rotate - use local_salts_prefix properly

https://gerrit.wikimedia.org/r/679941

gerritbot added a comment.Apr 15 2021, 8:25 PM

Change 679941 merged by Ottomata:

[operations/puppet@production] refine_sanitize_salt_rotate - use local_salts_prefix properly

https://gerrit.wikimedia.org/r/679941

Ottomata mentioned this in T280293: Delete UpperCased eventlogging legacy directories in /wmf/data/event 90 days from 2021-04-15 (after 2021-07-14).Apr 15 2021, 8:45 PM
gerritbot added a comment.Apr 16 2021, 4:11 PM

Change 680382 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery/source@master] SanitizeTransformation - Just some simple logging improvements.

https://gerrit.wikimedia.org/r/680382

gerritbot added a comment.Apr 19 2021, 1:01 PM

Change 681069 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/refine_sanitize - absent sanitize_eventlogging_analytics_delayed_test until June

https://gerrit.wikimedia.org/r/681069

gerritbot added a comment.Apr 19 2021, 1:07 PM

Change 681069 merged by Ottomata:

[operations/puppet@production] test/refine_sanitize - absent sanitize delayed_test until June

https://gerrit.wikimedia.org/r/681069

Ottomata updated the task description. (Show Details)Apr 19 2021, 1:46 PM
Ottomata updated the task description. (Show Details)Apr 19 2021, 1:52 PM
gerritbot added a comment.Apr 19 2021, 2:04 PM

Change 681092 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] analytics test - import mediawiki.page-data with camus and refine it

https://gerrit.wikimedia.org/r/681092

gerritbot added a comment.Apr 19 2021, 2:14 PM

Change 681092 merged by Ottomata:

[operations/puppet@production] analytics test - import mediawiki.page-delete with camus and refine it

https://gerrit.wikimedia.org/r/681092

gerritbot added a comment.Apr 19 2021, 2:39 PM

Change 681099 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] event_sanitized_allowlist - mediawiki_page_delete: keep_all

https://gerrit.wikimedia.org/r/681099

gerritbot added a comment.Apr 19 2021, 2:53 PM

Change 680382 merged by Ottomata:

[analytics/refinery/source@master] SanitizeTransformation - Just some simple logging improvements.

https://gerrit.wikimedia.org/r/680382

gerritbot added a comment.Apr 19 2021, 2:58 PM

Change 679961 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/refine_sanitized - add a general purpose event_sanitized job

https://gerrit.wikimedia.org/r/679961

gerritbot added a comment.Apr 19 2021, 3:01 PM

Change 681105 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/refine_sanitized - add a general purpose event_sanitized job

https://gerrit.wikimedia.org/r/681105

gerritbot added a comment.Apr 19 2021, 3:02 PM

Change 679961 abandoned by Ottomata:

[operations/puppet@production] test/refine_sanitized - add a general purpose event_sanitized job

Reason:

Iae3a76a37169a852e9698157b6b06e5b1e99f47a

https://gerrit.wikimedia.org/r/679961

gerritbot added a comment.Apr 20 2021, 1:50 PM

Change 681099 merged by Ottomata:

[analytics/refinery@master] event_sanitized_main_allowlist - mediawiki_page_delete: keep_all

https://gerrit.wikimedia.org/r/681099

gerritbot added a comment.Apr 20 2021, 3:32 PM

Change 681105 merged by Ottomata:

[operations/puppet@production] test/refine_sanitized - add a event_sanitized_main job

https://gerrit.wikimedia.org/r/681105

Ottomata mentioned this in T280813: Rename event_sanitized partition directories to lowercase.Apr 21 2021, 3:57 PM
gerritbot added a comment.Apr 21 2021, 6:42 PM

Change 678941 merged by Ottomata:

[operations/puppet@production] Move sanitize_eventlogging_analytics jobs from data_purge to refine_santizie

https://gerrit.wikimedia.org/r/678941

gerritbot added a comment.Apr 21 2021, 9:26 PM

Change 681784 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] sanitize_eventlogging_analytics_immediate - ensure absent during switchover

https://gerrit.wikimedia.org/r/681784

gerritbot added a comment.Apr 21 2021, 9:28 PM

Change 681784 merged by Ottomata:

[operations/puppet@production] sanitize_eventlogging_analytics_immediate - ensure absent during switchover

https://gerrit.wikimedia.org/r/681784

gerritbot added a comment.Apr 22 2021, 2:37 PM

Change 681991 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] refine_sanitize - use refinery 0.1.6 with RefineSanitize job

https://gerrit.wikimedia.org/r/681991

gerritbot added a comment.Apr 22 2021, 4:27 PM

Change 681991 merged by Ottomata:

[operations/puppet@production] refine_sanitize - use refinery 0.1.6 with RefineSanitize job

https://gerrit.wikimedia.org/r/681991

Ottomata updated the task description. (Show Details)Apr 23 2021, 4:19 AM
gerritbot added a comment.Apr 23 2021, 4:52 PM

Change 682175 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] refine - allow configuring RefineMonitors since and until params

https://gerrit.wikimedia.org/r/682175

gerritbot added a comment.Apr 23 2021, 5:05 PM

Change 682175 merged by Ottomata:

[operations/puppet@production] refine - allow configuring RefineMonitors since and until params

https://gerrit.wikimedia.org/r/682175

gerritbot added a comment.Apr 27 2021, 4:15 PM

Change 682986 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Rename and symlink sanitization eventlogging/whitelist.yaml

https://gerrit.wikimedia.org/r/682986

gerritbot added a comment.Apr 27 2021, 7:18 PM

Change 683043 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Add more tables to sanitize in event_sanitized_main_allowlist

https://gerrit.wikimedia.org/r/683043

Ottomata mentioned this in T281300: Drop old WMDEBanner events from Hive.Apr 27 2021, 7:39 PM
gerritbot added a comment.Apr 27 2021, 8:06 PM

Change 683053 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/data_purge - add drop_event job

https://gerrit.wikimedia.org/r/683053

gerritbot added a comment.Apr 27 2021, 8:53 PM

Change 682986 merged by Ottomata:

[analytics/refinery@master] Rename and symlink sanitization eventlogging/whitelist.yaml

https://gerrit.wikimedia.org/r/682986

gerritbot added a comment.Apr 27 2021, 8:53 PM

Change 683043 merged by Ottomata:

[analytics/refinery@master] Add more tables to sanitize in event_sanitized_main_allowlist

https://gerrit.wikimedia.org/r/683043

gerritbot added a comment.Apr 28 2021, 12:45 PM

Change 683053 merged by Ottomata:

[operations/puppet@production] test/data_purge - add drop_event job

https://gerrit.wikimedia.org/r/683053

Stashbot added a comment.Apr 28 2021, 12:50 PM

Mentioned in SAL (#wikimedia-analytics) [2021-04-28T12:50:06Z] <ottomata> applied data_purge jobs in analytics test cluster; old data will now be dropped there - T273789

gerritbot added a comment.Apr 28 2021, 1:12 PM

Change 683275 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery/source@master] RefineSanitizeMonitor - pass keep_all_enabled to SanitizeTransformation.loadAllowlist

https://gerrit.wikimedia.org/r/683275

gerritbot added a comment.Apr 28 2021, 1:28 PM

Change 683275 merged by Ottomata:

[analytics/refinery/source@master] RefineSanitizeMonitor - pass keep_all_enabled to SanitizeTransformation.loadAllowlist

https://gerrit.wikimedia.org/r/683275

Ottomata added a comment.EditedApr 28 2021, 2:06 PM

To prep for moving historical event data to event_sanitized, we need to copy the tables we'll be keeping forever there. Procedure:

# On an-launcher1002 in /home/otto
mkdir event_sanitized_main_T280813
cd event_sanitized_main_T280813

tables="
mediawiki_centralnotice_campaign_change \
mediawiki_centralnotice_campaign_create \
mediawiki_page_create \
mediawiki_page_delete \
mediawiki_page_links_change \
mediawiki_page_move \
mediawiki_page_restrictions_change \
mediawiki_page_suppress \
mediawiki_page_undelete \
mediawiki_revision_create \
mediawiki_revision_recommendation_create \
mediawiki_revision_score \
mediawiki_revision_tags_change \
mediawiki_revision_visibility_change \
mediawiki_user_blocks_change"

distcp_source_file=./distcp_source.txt
truncate -s 0 $distcp_source_file

for table in $tables; do
    path_orig="hdfs://analytics-hadoop/wmf/data/event/$table"
    path_new="hdfs://analytics-hadoop/wmf/data/event_sanitized/$table"

    # append the distcp source_file with path_orig.
    echo $path_orig >> $distcp_source_file

    # Build Hive scripts to create and add partitions to the event_sanitized table
    hive_sql_file=./${table}_create.hql
    truncate -s 0 $hive_sql_file

    # Build Hive scripts to recreate the tables with new lowercased partitions and then repair the tables to recreate the partitions.
    echo "SET mapred.child.java.opts=-Xmx8G -XX:+UseConcMarkSweepGC  -XX:-UseGCOverheadLimit;" >> $hive_sql_file
    echo "SET hive.msck.repair.batch.size=5000;" >> $hive_sql_file;
    echo "CREATE EXTERNAL TABLE IF NOT EXISTS event_sanitized.${table} LIKE event.${table};" >> $hive_sql_file
    echo "ALTER TABLE event_sanitized.${table} SET LOCATION '$path_new';" >> $hive_sql_file
    echo "MSCK REPAIR TABLE event_sanitized.${table};" >> $hive_sql_file
done


# do it!
mkdir ./done

# First run the distcp to copy all table directories into /wmf/data/event_sanitized
sudo -u analytics kerberos-run-command analytics hdfs dfs -put $distcp_source_file /tmp/distcp_event_sanitized_main_T280813.txt
time sudo -u analytics kerberos-run-command analytics hadoop distcp -m 60 -p -f /tmp/distcp_event_sanitized_main_T280813.txt hdfs://analytics-hadoop/wmf/data/event_sanitized
mv $distcp_source_file ./done/

for table in $tables; do
    echo "Creating and repairing event_sanitized.$table"
    time sudo -u analytics kerberos-run-command analytics hive -f ./${table}_create.hql && mv -v ./${table}_create.hql ./done/
    echo ""
done
gerritbot added a comment.Apr 28 2021, 4:43 PM

Change 683375 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Update refine_sanitize job to use event_sanitized_analytics_allowlist

https://gerrit.wikimedia.org/r/683375

gerritbot added a comment.Apr 28 2021, 4:44 PM

Change 683376 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] canary_events and refine_sanitize - use refinery 0.1.9

https://gerrit.wikimedia.org/r/683376

gerritbot added a comment.Apr 28 2021, 4:45 PM

Change 683375 merged by Ottomata:

[operations/puppet@production] Update refine_sanitize job to use event_sanitized_analytics_allowlist

https://gerrit.wikimedia.org/r/683375

gerritbot added a comment.Apr 28 2021, 4:47 PM

Change 683376 merged by Ottomata:

[operations/puppet@production] canary_events and refine_sanitize - use refinery 0.1.9

https://gerrit.wikimedia.org/r/683376

gerritbot added a comment.Apr 28 2021, 7:17 PM

Change 683421 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Update event_sanitized_main_allowlist

https://gerrit.wikimedia.org/r/683421

Ottomata added a comment.Apr 28 2021, 7:21 PM

For reference, the current sizes of the tables to move to event_sanitized are:

9.3 M  /wmf/data/event/mediawiki_centralnotice_campaign_change
1.4 M  /wmf/data/event/mediawiki_centralnotice_campaign_create
39.7 G  /wmf/data/event/mediawiki_page_create
4.4 G  /wmf/data/event/mediawiki_page_delete
309.5 G  /wmf/data/event/mediawiki_page_links_change
2.5 G  /wmf/data/event/mediawiki_page_move
2.5 K  /wmf/data/event/mediawiki_page_properties_change
428.5 M  /wmf/data/event/mediawiki_page_restrictions_change
10.4 M  /wmf/data/event/mediawiki_page_suppress
375.9 M  /wmf/data/event/mediawiki_page_undelete
392.1 G  /wmf/data/event/mediawiki_revision_create
13.9 M  /wmf/data/event/mediawiki_revision_recommendation_create
212.6 G  /wmf/data/event/mediawiki_revision_score
123.5 G  /wmf/data/event/mediawiki_revision_tags_change
1.1 G  /wmf/data/event/mediawiki_revision_visibility_change
3.8 G  /wmf/data/event/mediawiki_user_blocks_change
4.1 T  /wmf/data/event/resource_change
Ottomata added a comment.Apr 28 2021, 8:06 PM

FYI: We are not going to sanitize and keep resource_change

Ottomata added a comment.EditedApr 29 2021, 1:16 PM

distcp complete, application_id: application_1619507802557_6586

DistCp Counters
        Bytes Copied=1170743820170
        Bytes Expected=1170743820170
        Files Copied=1955833
        DIR_COPY=331299

Moving on to creating and repairing hive tables.

gerritbot added a comment.Apr 29 2021, 1:17 PM

Change 683421 merged by Ottomata:

[analytics/refinery@master] Update event_sanitized_main_allowlist

https://gerrit.wikimedia.org/r/683421

Stashbot added a comment.Apr 29 2021, 1:37 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-29T13:37:36Z] <otto@deploy1002> Started deploy [analytics/refinery@b3c5820]: update event_sanitized_main allowlst on an-launcher1002 - T273789

Stashbot added a comment.Apr 29 2021, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-29T13:40:35Z] <otto@deploy1002> Finished deploy [analytics/refinery@b3c5820]: update event_sanitized_main allowlst on an-launcher1002 - T273789 (duration: 02m 59s)

gerritbot added a comment.Apr 29 2021, 1:42 PM

Change 683614 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Enable event_sanitized_main_immediate job

https://gerrit.wikimedia.org/r/683614

gerritbot added a comment.Apr 29 2021, 1:56 PM

Change 683620 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery@master] Add comment in allowlist about mediawiki_page_properties_change

https://gerrit.wikimedia.org/r/683620

gerritbot added a comment.Apr 29 2021, 1:56 PM

Change 683620 merged by Ottomata:

[analytics/refinery@master] Add comment in allowlist about mediawiki_page_properties_change

https://gerrit.wikimedia.org/r/683620

gerritbot added a comment.Apr 29 2021, 1:58 PM

Change 683622 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Add comment in refine about mediawiki_page_properties_change

https://gerrit.wikimedia.org/r/683622

• fdans closed subtask T280813: Rename event_sanitized partition directories to lowercase as Resolved.Apr 29 2021, 2:55 PM
gerritbot added a comment.Apr 29 2021, 3:36 PM

Change 683614 merged by Ottomata:

[operations/puppet@production] Enable event_sanitized_main_immediate job

https://gerrit.wikimedia.org/r/683614

gerritbot added a comment.Apr 29 2021, 3:36 PM

Change 683622 merged by Ottomata:

[operations/puppet@production] Add comment in refine about mediawiki_page_properties_change

https://gerrit.wikimedia.org/r/683622

Stashbot added a comment.Apr 29 2021, 3:38 PM

Mentioned in SAL (#wikimedia-analytics) [2021-04-29T15:38:10Z] <ottomata> enabling event_sanitized_main jobs - T273789

Ottomata updated the task description. (Show Details)Apr 29 2021, 3:38 PM
Stashbot added a comment.Apr 29 2021, 4:15 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-29T16:15:37Z] <otto@deploy1002> Started deploy [analytics/refinery@b3c5820] (hadoop-test): update event_sanitized_main allowlst on an-launcher1002 - T273789

Stashbot added a comment.Apr 29 2021, 4:18 PM

Mentioned in SAL (#wikimedia-operations) [2021-04-29T16:18:17Z] <otto@deploy1002> Finished deploy [analytics/refinery@b3c5820] (hadoop-test): update event_sanitized_main allowlst on an-launcher1002 - T273789 (duration: 02m 39s)

Ottomata added a comment.Apr 29 2021, 4:25 PM

Backfilling since yesterday's distcp was started.

sudo -u analytics kerberos-run-command analytics /usr/local/bin/refine_event_sanitized_main_immediate --since=72

gerritbot added a comment.Apr 29 2021, 6:44 PM

Change 683698 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Remove refinery-drop-webrequest-sampled-druid job from test cluster

https://gerrit.wikimedia.org/r/683698

gerritbot added a comment.Apr 29 2021, 6:47 PM

Change 683698 merged by Ottomata:

[operations/puppet@production] Remove refinery-drop-webrequest-sampled-druid job from test cluster

https://gerrit.wikimedia.org/r/683698

gerritbot added a comment.Apr 30 2021, 1:02 PM

Change 683890 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Remove absented druid data drop job from test cluster

https://gerrit.wikimedia.org/r/683890

gerritbot added a comment.Apr 30 2021, 1:24 PM

Change 683890 merged by Ottomata:

[operations/puppet@production] Remove absented druid data drop job from test cluster

https://gerrit.wikimedia.org/r/683890

Ottomata added a comment.Apr 30 2021, 1:52 PM

ah, I need to reset the _REFINED flag mtimes for the hours I distcp-ed over, just like I did in https://phabricator.wikimedia.org/T280813#7028014. There's more data this time. Here's what I'm running now:


import org.wikimedia.analytics.refinery.job.refine._
import com.github.nscala_time.time.Imports._
import org.joda.time.format.ISODateTimeFormat
import org.apache.hadoop.fs.Path
import org.joda.time.Days

val isoDateTimeFormatter = ISODateTimeFormat.dateTimeParser()

// Have to split this into smaller chunks; default non yarn spark shell couldn't handle all partitions in this range at once.
val begin = DateTime.parse("2021-03-14T00:00:00Z", isoDateTimeFormatter)
val end = DateTime.parse("2021-05-01T00:00:00Z", isoDateTimeFormatter)


def daysInBetween(d1: DateTime, d2: DateTime): Seq[DateTime] = {
    val oldestDay = new DateTime(d1, DateTimeZone.UTC).hourOfDay.roundCeilingCopy
    val youngestDay = new DateTime(d2, DateTimeZone.UTC).hourOfDay.roundFloorCopy

    for (d <- 0 until Days.daysBetween(oldestDay, youngestDay).getDays) yield {
        oldestDay + d.days
    }
}

val days = daysInBetween(begin, end)
val outputPath = new Path("/wmf/data/event_sanitized")

for (i <- 0 until days.size - 1) yield {
    val since = days(i)
    val until = days(i + 1)

    println("==== ", since, until, " ====")

    val rts = RefineTarget.find(
        spark,
        "event",
        outputPath,
        "event_sanitized",
        since,
        until,
        None,
        None
    )

    val doneRts = rts.filter(_.doneFlagExists).par
    doneRts.foreach(_.writeDoneFlag)
    println("==== DONE ", since, until, doneRts.size, " ====")
    println("")
}
Ottomata added a comment.Apr 30 2021, 2:55 PM

Done.

Ottomata moved this task from Backlog to To be Estimated/To be discussed on the Event-Platform board.Apr 30 2021, 7:37 PM
gerritbot added a comment.May 3 2021, 2:55 PM

Change 684427 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/puppet@production] analytics:refinery:job:test:data_purge: remove -skipTrash from drop_event

https://gerrit.wikimedia.org/r/684427

gerritbot added a comment.May 3 2021, 3:01 PM

Change 684427 merged by Ottomata:

[operations/puppet@production] analytics:refinery:job:test:data_purge: remove -skipTrash from drop_event

https://gerrit.wikimedia.org/r/684427

gerritbot added a comment.May 12 2021, 2:55 PM

Change 689925 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/puppet@production] analytics:refinery:job:data_purge: remove -skipTrash from drop_event

https://gerrit.wikimedia.org/r/689925

Stashbot added a comment.May 12 2021, 3:17 PM

Mentioned in SAL (#wikimedia-analytics) [2021-05-12T15:17:39Z] <ottomata> dropped event.mediawiki_job_* tables and data directories with mforns - T273789

gerritbot added a comment.May 12 2021, 8:36 PM

Change 689925 merged by Ottomata:

[operations/puppet@production] analytics:refinery:job:data_purge: improve drop-el-unsanitized-events

https://gerrit.wikimedia.org/r/689925

gerritbot added a comment.May 12 2021, 8:50 PM

Change 690038 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] data_purge - Refactor drop-el-unsanitized-events in to drop_event job

https://gerrit.wikimedia.org/r/690038

gerritbot added a comment.May 12 2021, 8:52 PM

Change 690038 merged by Ottomata:

[operations/puppet@production] data_purge - Refactor drop-el-unsanitized-events in to drop_event job

https://gerrit.wikimedia.org/r/690038

Stashbot added a comment.May 12 2021, 8:57 PM

Mentioned in SAL (#wikimedia-operations) [2021-05-12T20:57:13Z] <ottomata> starting new drop_event data purge job to drop all event data older than 90 days in the Hive event database - T273789

Ottomata updated the task description. (Show Details)May 12 2021, 9:03 PM
Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.May 12 2021, 9:06 PM
Ottomata closed this task as Resolved.May 18 2021, 4:53 PM
gerritbot added a comment.Jul 13 2021, 7:29 PM

Change 675936 abandoned by Ottomata:

[operations/puppet@production] refine: rename EventLoggingSanitization to RefineSanitize

Reason:

Fixed elsewhere

https://gerrit.wikimedia.org/r/675936

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL