How to add a delete button to a PHP form that will delete a row from a MySQL table

You have to pass a variable in the delete link. You have to pass <?php echo $contact['name']; ?> (the name value) in a hidden field or pass this value in URL:

Replace

<td class="contact-delete">
      <form action='delete.php' method="post">
      <input type="hidden" name="name" value="">
      <input type="submit" name="submit" value="Delete">
      </form>
</td>

With

<td class="contact-delete">
    <form action='delete.php?name="<?php echo $contact['name']; ?>"' method="post">
        <input type="hidden" name="name" value="<?php echo $contact['name']; ?>">
        <input type="submit" name="submit" value="Delete">
    </form>
</td>