Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.

Device: MacBook Pro 16 M1 Max, 64GB running MacOS 12.0.1.

I tried setting up GPU Accelerated TensorFlow on my Mac using the following steps:

  1. Setup: XCode CLI / Homebrew/ Miniforge
  2. Conda Env: Python 3.9.5
  3. conda install -c apple tensorflow-deps
  4. python -m pip install tensorflow-macos
  5. python -m pip install tensorflow-metal
  6. brew install libjpeg
  7. conda install -y matplotlib jupyterlab
  8. In Jupyter Lab, I try to execute this code:
from tensorflow.keras import layers
from tensorflow.keras import models
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))
model.summary()

The code executes, but I get this warning, indicating no GPU Acceleration can be used as it defaults to a 0MB GPU. Error:

Metal device set to: Apple M1 Max
2021-10-27 08:23:32.872480: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2021-10-27 08:23:32.872707: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)

Anyone has any idea how to fix this? I came across a bunch of posts around here related to the same issue but with no solid fix. I created a new question as I found the other questions less descriptive of the issue, and wanted to comprehensively depict it. Any fix would be of much help.

Post not yet marked as solved Up vote post of suprateembanerjee Down vote post of suprateembanerjee
29k views
Posted by
suprateembanerjee
Reply

  • Metal device set to: Apple M1

    systemMemory: 16.00 GB maxCacheSize: 5.33 GB

    2021-12-13 19:59:56.135942: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2021-12-13 19:59:56.136049: I tensorflow/core/common_runtime/pluggåable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: )

    —
    JIAOJIAO-MEI
Add a Comment

Replies

Having the same issue here, on macbook pro m1 16gb & monterey

Posted by
clecarosc
Up vote reply of clecarosc Down vote reply of clecarosc
Add a Comment

It was working with me on Miniforg3 env, after moving to the other environment (Anaconda3) to do some work of torch and back again to my arm64 environment it shows the same message and didn't work, I made the below steps and it back to work again.

  1. I uninstalled the tensorflow from all environments.
  2. Uninstalled tensorflow-metal and tensorflow-deps
  3. Restart the device.
  4. Reinstalled the packages again.

Thanks.

Posted by
IbrahimSalameh
Up vote reply of IbrahimSalameh Down vote reply of IbrahimSalameh
Add a Comment

Changing from v0.4.0 to v.0.1.1 worked for me

pip uninstall tensorflow-me
pip install tensorflow-metal==0.1.1
Posted by
arthur-fontaine
Up vote reply of arthur-fontaine Down vote reply of arthur-fontaine

  • Did you mean 'pip uninstall tensorflow-metal'?

    Also, do we need to uninstall it before installing a specific version?

    —
    nzaw96
Add a Comment

I tried : pip install tensorflow-metal==0.1.1 but it cause different problem... :(

Init Plugin

2022-05-15 14:00:41.859860: F tensorflow/c/c_api_experimental.cc:739] Non-OK-status: tensorflow::RegisterPluggableDevicePlugin(lib_handle->lib_handle) status: FAILED_PRECONDITION: 'host_callback' field in SP_StreamExecutor must be set.

zsh: abort      python

Posted by
ZviStein
Up vote reply of ZviStein Down vote reply of ZviStein
Add a Comment

Try adding the following:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
Posted by
a0dh
Post not yet marked as solved Up vote reply of a0dh Down vote reply of a0dh
Add a Comment

Same error here when I do scheduling learning rate, jupyterlab dies

Posted by
Rayrayray
Up vote reply of Rayrayray Down vote reply of Rayrayray
Add a Comment

Changing the version of tensorflow-metal didn't help. It made my jupyter kernel restart. This is why I use pytorch. It works.

Posted by
MikeBee
Up vote reply of MikeBee Down vote reply of MikeBee
Add a Comment

I'm having the same issue, and, sometimes my training just suddenly stops in a random epoch. So now, in order to work, I have to use the parameter with tf.device('cpu:0'), to force CPU use instead of GPU. The worse part is that apple seems like it don't care about fixing that stuffs.

Posted by
LuizHDRamos
Up vote reply of LuizHDRamos Down vote reply of LuizHDRamos
Add a Comment

Apple Silicon is UMA, Unified Memory Architecture. This annoying message is just telling you that you are running on Apple Silicon. So ignore it.

Posted by
MikeBee
Post not yet marked as solved Up vote reply of MikeBee Down vote reply of MikeBee

  • I would be happy to ignore this error message, the problem is that model.fit gets totally stucked, it doesn't even start traning. NADA. If i switch to CPU, then everything works fine. Switch back to GPU, it prints out that error message and then it gets stuck on model.fit. The whole internet is flooded with this exact same problem and neither Apple or Tensoflow takes it seriously. If somone would properly classify this as a bug, then we would know it will get solved. Back to PC i guess!?

    —
    te_neon

  • I can't tell if the Ultra's GPU is really accelerating the fit or not. The GPU History in Activity Monitor only seems to spike if I scroll the notebook (or for example, the window I'm typing this comment into.)

    Ultimately am I better off running it on the M1 Ultra CPU, as some have suggested, than the GPUs that doubled the machine's cost?

    —
    maxhirez
Add a Comment

Why can't this be marked as solved so the answer can appear at the top? The M1 is a UMA device -- not a NUMA. A central point of this SOC is its unified memory. This message just says that Tensorflow recognizes that it's not a non-unified memory architecture system. Then it chooses the appropriate UMA algorithms. How does this forum help if answers are buried in noise? Stack Overflow does a better job.

Posted by
MikeBee
Post not yet marked as solved Up vote reply of MikeBee Down vote reply of MikeBee

  • That makes sense.

    —
    Elim_MA

  • Don't forget to upvote.

    —
    maxhirez
Add a Comment

FOR M1 ULTRA (128GB RAM, 20c CPU, 64c GPU) on MacOS 12.5, getting the following message:

Metal device set to: Apple M1 Ultra systemMemory: 128.00 GB maxCacheSize: 48.00 GB

2022-07-22 16:44:43.488061: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.

2022-07-22 16:44:43.488273: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: )

My question is: is Why is this error coming at all? Why NUMA? Moreover, GPU has 0MB memory? How is this possible?

Python: 3.9.13 tensorflow-macos: 2.9.2 tensorflow-metal: 0.5.0

Please help. Thanks, Bapi

Posted by
karbapi
Post not yet marked as solved Up vote reply of karbapi Down vote reply of karbapi

  • Getting the exact same messages with Python: 3.8.13 tensorflow-macos: 2.8.0 tensorflow-metal: 0.4.0. Moreover, there is a "tangible time gap between the epochs".

    —
    karbapi
Add a Comment

I intended to speedup the training process. now what is this (got during training with workers=8, use_multiprocessing=True)? STRANGE!!!! Never got it with my MBP-13 (2017, i5 core, 16GB RAM) with the same code.

Traceback (most recent call last):   File "", line 1, in   File "/Users/bapikar/miniforge3/envs/tf28_python38/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main     exitcode = _main(fd, parent_sentinel)   File "/Users/bapikar/miniforge3/envs/tf28_python38/lib/python3.8/multiprocessing/spawn.py", line 126, in _main     self = reduction.pickle.load(from_parent)   File "/Users/bapikar/miniforge3/envs/tf28_python38/lib/python3.8/multiprocessing/synchronize.py", line 110, in setstate     self._semlock = _multiprocessing.SemLock._rebuild(*state) FileNotFoundError: [Errno 2] No such file or directory

Posted by
karbapi
Up vote reply of karbapi Down vote reply of karbapi

  • any update on this?

    —
    karbapi

  • I have the exact same problem.

    Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.

    Some people actually think it is just a warning. It is NOT a warning, training doesn't even get started, i gets STUCK, even if i leave it for 30 minutes. When i switch to CPU, it works fine. But isn't the whole purpose of high-performance GPU to train model on GPU? Apple support, are you reading this? Tensorflow does NOT work on GPU on M1 !!!

    —
    te_neon
Add a Comment

I bought a new M1 Macbook pro M1 and I was hoping i could use it for machine learning on GPU (with metal). But i it doesn't work on GPU at all!! It gets stuck on model.fit with the famous error message "NUMA node of platform GPU ID 0". And if i switch to CPU, then it does work actually, but the point of (deep) machine learning is to utilize the GPU, to make training faster, right?! Apple, please help us, fix this issue, take it seriously.

Posted by
te_neon
Post not yet marked as solved Up vote reply of te_neon Down vote reply of te_neon

  • my 64c GPU is sitting IDLE too! what METAL is it? :-(

    —
    karbapi
Add a Comment

Was having the same issue. script would crash after message, bus error. I'm using Mac Pro (Late 2013) AMD FirePro D500 3 GB

What fixed it for me was, in my tensorflow-metal virtual env, I changed my version $pip install =Iv tensorflow-metal==0.60

Then inside my script set the following, after importing os os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'

After this I ran the script and it worked and in Activity Monitor I could see it uses the GPU

Posted by
wsimpson2019
Up vote reply of wsimpson2019 Down vote reply of wsimpson2019
Add a Comment

Same problem on the M1 MacBook Air with Mac OS 12.6 (Monetary) 2023-01-24 01:32:00.798327: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:306] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2023-01-24 01:32:00.798694: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>) WARNING:tensorflow:AutoGraph could not transform <function normalize_img at 0x16a452cb0> and will run it as-is. Cause: Unable to locate the source code of <function normalize_img at 0x16a452cb0>. Note that functions defined in certain environments, like the interactive Python shell, do not expose their source code. If that is the case, you should define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.experimental.do_not_convert. Original error: could not get source code To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert

Posted by
ClementTong
Up vote reply of ClementTong Down vote reply of ClementTong
Add a Comment