WARNING 2022-03-18 09:54:55,215 launch.py:423] Not found distinct arguments and compiled with cuda or xpu. Default use collective mode INFO 2022-03-18 09:54:55,215 launch_utils.py:641] Change selected_gpus into reletive values. --ips:0,1,2,3 will change into relative_ips:[0, 1, 2, 3] according to your CUDA_VISIBLE_DEVICES:['0', '1', '2', '3'] INFO 2022-03-18 09:54:55,216 launch_utils.py:528] Local start 4 processes. First process distributed environment info (Only For Debug): +=======================================================================================+ | Distributed Envs Value | +---------------------------------------------------------------------------------------+ | PADDLE_TRAINER_ID 0 | | PADDLE_CURRENT_ENDPOINT 127.0.0.1:60008 | | PADDLE_TRAINERS_NUM 4 | | PADDLE_TRAINER_ENDPOINTS ... 0.1:60009,127.0.0.1:60010,127.0.0.1:60011| | PADDLE_RANK_IN_NODE 0 | | PADDLE_LOCAL_DEVICE_IDS 0 | | PADDLE_WORLD_DEVICE_IDS 0,1,2,3 | | FLAGS_selected_gpus 0 | | FLAGS_selected_accelerators 0 | +=======================================================================================+ INFO 2022-03-18 09:54:55,216 launch_utils.py:532] details abouts PADDLE_TRAINER_ENDPOINTS can be found in log/endpoints.log, and detail running logs maybe found in log/workerlog.0 ----------- Configuration Arguments ----------- backend: auto elastic_server: None force: False gpus: 0,1,2,3 heter_devices: heter_worker_num: None heter_workers: host: None http_port: None ips: 127.0.0.1 job_id: None log_dir: log np: None nproc_per_node: None run_mode: None scale: 0 server_num: None servers: training_script: tools/train.py training_script_args: ['-c', 'configs/picodet/picodet_xs_320_coco.yml', '--eval'] worker_num: None workers: ------------------------------------------------ launch train in GPU mode! launch proc_id:31530 idx:0 launch proc_id:31533 idx:1 launch proc_id:31536 idx:2 launch proc_id:31539 idx:3 Warning: import ppdet from source directory without installing, run 'python setup.py install' to install ppdet firstly server not ready, wait 3 sec to retry... not ready endpoints:['127.0.0.1:60009', '127.0.0.1:60011'] I0318 09:55:02.421108 31530 nccl_context.cc:74] init nccl context nranks: 4 local rank: 0 gpu id: 0 ring id: 0 W0318 09:55:03.428002 31530 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 10.1 W0318 09:55:03.432008 31530 device_context.cc:465] device: 0, cuDNN Version: 7.6. loading annotations into memory... Done (t=19.09s) creating index... index created! [03/18 09:55:33] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 200365, area: 0.0 x1: 296.65, y1: 388.33, x2: 297.67999999999995, y2: 388.33. [03/18 09:55:47] ppdet.data.source.coco WARNING: Found an invalid bbox in annotations: im_id: 550395, area: 0.0 x1: 9.98, y1: 188.56, x2: 15.52, y2: 188.56. [03/18 09:55:52] ppdet.utils.download INFO: Downloading LCNet_x0_35_pretrained.pdparams from https://paddledet.bj.bcebos.com/models/pretrained/LCNet_x0_35_pretrained.pdparams 0%| | 0/6447 [00:00